No runs match your search.
Environment: PROD
| Time (UTC) | Job | Failed at | Result | PR | Failed tests | Details |
|---|---|---|---|---|---|---|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
e2e, unknown | failed | n/a | 15 | e2e (14)timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
<*fmt.wrapErrors | 0xc0008c0420>:
failed waiting for hcpCluster="cluster-zstream-4-19-jrgxnr" in resourcegroup="rg-zstream-upgrade-4-19-5mbcpg" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-jrgxnr in resource group rg-zstream-upgrade-4-19-5mbcpg, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-jrgxnr\" in resourcegroup=\"rg-zstream-upgrade-4-19-5mbcpg\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-jrgxnr in resource group rg-zstream-upgrade-4-19-5mbcpg, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0007fd0b0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-jrgxnr in resource group rg-zstream-upgrade-4-19-5mbcpg",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
<*fmt.wrapErrors | 0xc0008c0420>:
failed waiting for hcpCluster="cluster-zstream-4-19-jrgxnr" in resourcegroup="rg-zstream-upgrade-4-19-5mbcpg" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-jrgxnr in resource group rg-zstream-upgrade-4-19-5mbcpg, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-jrgxnr\" in resourcegroup=\"rg-zstream-upgrade-4-19-5mbcpg\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-jrgxnr in resource group rg-zstream-upgrade-4-19-5mbcpg, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0007fd0b0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-jrgxnr in resource group rg-zstream-upgrade-4-19-5mbcpg",
},
<context.deadlineExceededError>{},
],
}
occurredERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:118]: Unexpected error:
<*fmt.wrapError | 0xc000696dc0>:
failed creating external auth "ea-list" in resourcegroup="ea-list-zdqkwr" for cluster="ea-listmrjvrf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-zdqkwr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listmrjvrf/externalAuths/ea-list
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed creating external auth \"ea-list\" in resourcegroup=\"ea-list-zdqkwr\" for cluster=\"ea-listmrjvrf\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-zdqkwr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listmrjvrf/externalAuths/ea-list\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0015f1560>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"X-Ms-Request-Id": [
"166169b8-0520-43f1-8408-4075317b3e22",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260402T234332Z:cbd8afe4-1516-446c-bfca-e27700abfeb7",
],
"X-Content-Type-Options": ["nosniff"],
"X-Envoy-Upstream-Service-Time": ["238"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/0aa7dca7-ac55-417f-a9ed-45134e5db709",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Msedge-Ref": [
"Ref A: A7DD4C4BA3564926A620C3D8ADA15FE4 Ref B: CHI261140511054 Ref C: 2026-04-02T23:43:32Z",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Correlation-Request-Id": [
"cbd8afe4-1516-446c-bfca-e27700abfeb7",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Content-Length": ["107"],
"Content-Type": ["application/json"],
"Expires": ["-1"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Thu, 02 Apr 2026 23:43:32 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000696d00>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:118]: Unexpected error:
<*fmt.wrapError | 0xc000696dc0>:
failed creating external auth "ea-list" in resourcegroup="ea-list-zdqkwr" for cluster="ea-listmrjvrf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-zdqkwr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listmrjvrf/externalAuths/ea-list
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed creating external auth \"ea-list\" in resourcegroup=\"ea-list-zdqkwr\" for cluster=\"ea-listmrjvrf\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-zdqkwr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listmrjvrf/externalAuths/ea-list\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0015f1560>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"X-Ms-Request-Id": [
"166169b8-0520-43f1-8408-4075317b3e22",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260402T234332Z:cbd8afe4-1516-446c-bfca-e27700abfeb7",
],
"X-Content-Type-Options": ["nosniff"],
"X-Envoy-Upstream-Service-Time": ["238"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/0aa7dca7-ac55-417f-a9ed-45134e5db709",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Msedge-Ref": [
"Ref A: A7DD4C4BA3564926A620C3D8ADA15FE4 Ref B: CHI261140511054 Ref C: 2026-04-02T23:43:32Z",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Correlation-Request-Id": [
"cbd8afe4-1516-446c-bfca-e27700abfeb7",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Content-Length": ["107"],
"Content-Type": ["application/json"],
"Expires": ["-1"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Date": [
"Thu, 02 Apr 2026 23:43:32 GMT",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000696d00>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
Opaque: "",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredDeserializaion Error: no output from command Show raw failureRunning Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775174183 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231
STEP: creating resource group @ 04/03/26 00:04:23.463
"ts"="2026-04-03 00:04:23.463319" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-7s5scz-b2nc6w"
STEP: creating cluster parameters at control plane version @ 04/03/26 00:04:27.591
STEP: creating customer resources @ 04/03/26 00:04:27.714
STEP: creating the HCP cluster with version 4.21.7 @ 04/03/26 00:05:20.496
"ts"="2026-04-03 00:05:20.496874" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-7s5scz" "resourceGroup"="rg-np-version-upgrade-7s5scz-b2nc6w"
STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/03/26 00:18:42.133
STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/03/26 00:30:11.111
STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/03/26 00:33:47.481
STEP: capturing node release images before upgrade @ 04/03/26 00:33:47.596
STEP: triggering nodepool upgrade to version 4.21.7 @ 04/03/26 00:33:47.644
STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/03/26 00:34:39.143
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775174183 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231
STEP: creating resource group @ 04/03/26 00:04:23.463
"ts"="2026-04-03 00:04:23.463319" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-7s5scz-b2nc6w"
STEP: creating cluster parameters at control plane version @ 04/03/26 00:04:27.591
STEP: creating customer resources @ 04/03/26 00:04:27.714
STEP: creating the HCP cluster with version 4.21.7 @ 04/03/26 00:05:20.496
"ts"="2026-04-03 00:05:20.496874" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-7s5scz" "resourceGroup"="rg-np-version-upgrade-7s5scz-b2nc6w"
STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/03/26 00:18:42.133
STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/03/26 00:30:11.111
STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/03/26 00:33:47.481
STEP: capturing node release images before upgrade @ 04/03/26 00:33:47.596
STEP: triggering nodepool upgrade to version 4.21.7 @ 04/03/26 00:33:47.644
STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/03/26 00:34:39.143
Command Error: signal: killed
Deserializaion Error: no output from commandERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:172]: Unexpected error:
<*fmt.wrapError | 0xc00138d620>:
failed creating external auth "external-auth" in resourcegroup="external-auth-cluster-9tvzvg" for cluster="ea-cluster": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-9tvzvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed creating external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-9tvzvg\" for cluster=\"ea-cluster\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-9tvzvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000985ad0>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Expires": ["-1"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"39fbee8f-3cf5-48e0-b960-f67dbe52bffb",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Msedge-Ref": [
"Ref A: 61F6A3FD78CC4CD8BFF56C31AA11F0E1 Ref B: CH1AA2020603031 Ref C: 2026-04-02T23:58:09Z",
],
"Date": [
"Thu, 02 Apr 2026 23:58:09 GMT",
],
"X-Envoy-Upstream-Service-Time": ["260"],
"X-Ms-Failure-Cause": ["service"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Pragma": ["no-cache"],
"Content-Length": ["107"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/90122db5-9edb-46e1-a78d-3e8a94c781aa",
],
"X-Ms-Correlation-Request-Id": [
"ad53fc85-9a2b-46de-b180-070eb50b5a68",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260402T235809Z:ad53fc85-9a2b-46de-b180-070eb50b5a68",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Content-Type-Options": ["nosniff"],
},
Body: <*exported.nopClosingBytesReader | 0xc00138d560>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:172]: Unexpected error:
<*fmt.wrapError | 0xc00138d620>:
failed creating external auth "external-auth" in resourcegroup="external-auth-cluster-9tvzvg" for cluster="ea-cluster": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-9tvzvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed creating external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-9tvzvg\" for cluster=\"ea-cluster\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-9tvzvg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000985ad0>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Expires": ["-1"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"39fbee8f-3cf5-48e0-b960-f67dbe52bffb",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Msedge-Ref": [
"Ref A: 61F6A3FD78CC4CD8BFF56C31AA11F0E1 Ref B: CH1AA2020603031 Ref C: 2026-04-02T23:58:09Z",
],
"Date": [
"Thu, 02 Apr 2026 23:58:09 GMT",
],
"X-Envoy-Upstream-Service-Time": ["260"],
"X-Ms-Failure-Cause": ["service"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Pragma": ["no-cache"],
"Content-Length": ["107"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/90122db5-9edb-46e1-a78d-3e8a94c781aa",
],
"X-Ms-Correlation-Request-Id": [
"ad53fc85-9a2b-46de-b180-070eb50b5a68",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260402T235809Z:ad53fc85-9a2b-46de-b180-070eb50b5a68",
],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Content-Type-Options": ["nosniff"],
},
Body: <*exported.nopClosingBytesReader | 0xc00138d560>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
<*fmt.wrapErrors | 0xc0015907b0>:
failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-tndx4h" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-tndx4h, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-tndx4h\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-tndx4h, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000e99a40>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-tndx4h",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
<*fmt.wrapErrors | 0xc0015907b0>:
failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-tndx4h" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-tndx4h, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-tndx4h\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-tndx4h, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000e99a40>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-tndx4h",
},
<context.deadlineExceededError>{},
],
}
occurredInterrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
<*fmt.wrapErrors | 0xc000a06c00>:
failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-p7j8mg" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-p7j8mg, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-p7j8mg\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-p7j8mg, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00067e680>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-p7j8mg",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
<*fmt.wrapErrors | 0xc000a06c00>:
failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-p7j8mg" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-p7j8mg, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-p7j8mg\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-p7j8mg, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00067e680>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-p7j8mg",
},
<context.deadlineExceededError>{},
],
}
occurredtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
<*fmt.wrapErrors | 0xc000a1cc60>:
failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-nm2pxw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nm2pxw, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-nm2pxw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nm2pxw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000a7e4e0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nm2pxw",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
<*fmt.wrapErrors | 0xc000a1cc60>:
failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-nm2pxw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nm2pxw, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-nm2pxw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nm2pxw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000a7e4e0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nm2pxw",
},
<context.deadlineExceededError>{},
],
}
occurredInterrupted by User Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:51]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:51]: Interrupted by User ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error:
<*fmt.wrapError | 0xc000a21680>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-7z8hbg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-7z8hbg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000845d40>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Envoy-Upstream-Service-Time": ["154"],
"X-Content-Type-Options": ["nosniff"],
"Date": [
"Thu, 02 Apr 2026 23:47:08 GMT",
],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"a6270c0d-5904-4199-be64-b4731e31f158",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/25b0928e-3d5f-45fb-ab35-db007159f40d",
],
"X-Ms-Correlation-Request-Id": [
"83a37837-14b6-447f-9ee2-d29ececcd65c",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260402T234708Z:83a37837-14b6-447f-9ee2-d29ececcd65c",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: 18AE25CEB3C847B2B0880A4661A960C2 Ref B: CHI261140511054 Ref C: 2026-04-02T23:47:08Z",
],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Content-Length": ["107"],
"Content-Type": ["application/json"],
},
Body: <*exported.nopClosingBytesReader | 0xc000a215c0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "ma...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error:
<*fmt.wrapError | 0xc000a21680>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-7z8hbg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-7z8hbg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000845d40>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Envoy-Upstream-Service-Time": ["154"],
"X-Content-Type-Options": ["nosniff"],
"Date": [
"Thu, 02 Apr 2026 23:47:08 GMT",
],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Request-Id": [
"a6270c0d-5904-4199-be64-b4731e31f158",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/25b0928e-3d5f-45fb-ab35-db007159f40d",
],
"X-Ms-Correlation-Request-Id": [
"83a37837-14b6-447f-9ee2-d29ececcd65c",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260402T234708Z:83a37837-14b6-447f-9ee2-d29ececcd65c",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Expires": ["-1"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: 18AE25CEB3C847B2B0880A4661A960C2 Ref B: CHI261140511054 Ref C: 2026-04-02T23:47:08Z",
],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"Content-Length": ["107"],
"Content-Type": ["application/json"],
},
Body: <*exported.nopClosingBytesReader | 0xc000a215c0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "ma...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredDeserializaion Error: no output from command Show raw failureRunning Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775173098 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36
STEP: creating a resource group @ 04/03/26 00:04:23.094
"ts"="2026-04-03 00:04:23.094924" "level"=0 "msg"="creating resource group" "resourceGroup"="np-autoscaling-d2zlbp"
STEP: creating cluster parameters @ 04/03/26 00:04:26.964
STEP: creating customer resources @ 04/03/26 00:04:26.964
STEP: creating the HCP cluster @ 04/03/26 00:05:20.217
"ts"="2026-04-03 00:05:20.217661" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-autoscale-cluster" "resourceGroup"="np-autoscaling-d2zlbp"
STEP: creating nodepool parameters with autoscaling enabled @ 04/03/26 00:18:41.499
STEP: creating the autoscaling nodepool @ 04/03/26 00:18:41.499
STEP: verifying the cluster has default autoscaling parameters @ 04/03/26 00:31:21.659
"ts"="2026-04-03 00:31:21.802840" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-03 00:31:21.802917" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-autoscale-cluster-pdldv9" "resourceGroup"="np-autoscaling-d2zlbp"
"ts"="2026-04-03 00:31:21.916762" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-autoscale-cluster-pdldv9" "resourceGroup"=""
"ts"="2026-04-03 00:31:22.918087" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
"ts"="2026-04-03 00:31:22.918195" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
"ts"="2026-04-03 00:31:22.918256" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-03 00:31:22.919731" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-132"
"ts"="2026-04-03 00:31:22.919749" "level"=0 "msg"="collecting deployments" "resourceGroup"="np-autoscaling-d2zlbp"
"ts"="2026-04-03 00:31:23.016046" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-d2zlbp" "deployment"="pooledNonMsiScopedAssignments-ssrwqrnhyyabs"
"ts"="2026-04-03 00:31:23.121326" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-132" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-03 00:31:23.128889" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-d2zlbp" "deployment"="customer-infra-np-autoscale-cluster-pdldv9"
"ts"="2026-04-03 00:31:23.288779" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-132" "deployment"="msi-deployment-132"
"ts"="2026-04-03 00:31:23.467072" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-03 00:31:23.467203" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-03 00:31:23.467319" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="np-autoscaling-d2zlbp"
"ts"="2026-04-03 00:50:52.078081" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="np-autoscaling-d2zlbp"
"ts"="2026-04-03 00:50:52.078111" "level"=0 "msg"="deleting resource group" "resourceGroup"="np-autoscaling-d2zlbp"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775173098 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36
STEP: creating a resource group @ 04/03/26 00:04:23.094
"ts"="2026-04-03 00:04:23.094924" "level"=0 "msg"="creating resource group" "resourceGroup"="np-autoscaling-d2zlbp"
STEP: creating cluster parameters @ 04/03/26 00:04:26.964
STEP: creating customer resources @ 04/03/26 00:04:26.964
STEP: creating the HCP cluster @ 04/03/26 00:05:20.217
"ts"="2026-04-03 00:05:20.217661" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-autoscale-cluster" "resourceGroup"="np-autoscaling-d2zlbp"
STEP: creating nodepool parameters with autoscaling enabled @ 04/03/26 00:18:41.499
STEP: creating the autoscaling nodepool @ 04/03/26 00:18:41.499
STEP: verifying the cluster has default autoscaling parameters @ 04/03/26 00:31:21.659
"ts"="2026-04-03 00:31:21.802840" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-03 00:31:21.802917" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-autoscale-cluster-pdldv9" "resourceGroup"="np-autoscaling-d2zlbp"
"ts"="2026-04-03 00:31:21.916762" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-autoscale-cluster-pdldv9" "resourceGroup"=""
"ts"="2026-04-03 00:31:22.918087" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
"ts"="2026-04-03 00:31:22.918195" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
"ts"="2026-04-03 00:31:22.918256" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-03 00:31:22.919731" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-132"
"ts"="2026-04-03 00:31:22.919749" "level"=0 "msg"="collecting deployments" "resourceGroup"="np-autoscaling-d2zlbp"
"ts"="2026-04-03 00:31:23.016046" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-d2zlbp" "deployment"="pooledNonMsiScopedAssignments-ssrwqrnhyyabs"
"ts"="2026-04-03 00:31:23.121326" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-132" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-03 00:31:23.128889" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-d2zlbp" "deployment"="customer-infra-np-autoscale-cluster-pdldv9"
"ts"="2026-04-03 00:31:23.288779" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-132" "deployment"="msi-deployment-132"
"ts"="2026-04-03 00:31:23.467072" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-03 00:31:23.467203" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-03 00:31:23.467319" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="np-autoscaling-d2zlbp"
"ts"="2026-04-03 00:50:52.078081" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="np-autoscaling-d2zlbp"
"ts"="2026-04-03 00:50:52.078111" "level"=0 "msg"="deleting resource group" "resourceGroup"="np-autoscaling-d2zlbp"
Command Error: signal: killed
Deserializaion Error: no output from commandDeserializaion Error: no output from command Show raw failureRunning Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775174130 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should be able to update node pool labels and taints [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
STEP: creating a resource group @ 04/02/26 23:55:30.12
"ts"="2026-04-02 23:55:30.120343" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-labels-taints-vvn7hc"
STEP: creating cluster parameters @ 04/02/26 23:55:34.115
STEP: creating customer resources @ 04/02/26 23:55:34.116
STEP: creating the HCP cluster @ 04/02/26 23:56:27.658
"ts"="2026-04-02 23:56:27.658606" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cluster-np-labels-taints" "resourceGroup"="rg-np-labels-taints-vvn7hc"
STEP: getting credentials @ 04/03/26 00:05:46.538
STEP: creating the node pool with initial labels and taints @ 04/03/26 00:07:53.808
STEP: waiting for nodes to be ready @ 04/03/26 00:16:31.838
STEP: verifying initial labels are present on nodes @ 04/03/26 00:16:31.986
STEP: verifying initial taints are present on nodes @ 04/03/26 00:16:32.013
STEP: updating nodepool with new taints and scaling up @ 04/03/26 00:16:32.013
STEP: verifying nodes are scaled to the expected count @ 04/03/26 00:29:03.022
STEP: verifying new taint is present on newly created node @ 04/03/26 00:29:03.167
STEP: updating nodepool with a new label and scaling up @ 04/03/26 00:29:03.194
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775174130 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should be able to update node pool labels and taints [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
STEP: creating a resource group @ 04/02/26 23:55:30.12
"ts"="2026-04-02 23:55:30.120343" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-labels-taints-vvn7hc"
STEP: creating cluster parameters @ 04/02/26 23:55:34.115
STEP: creating customer resources @ 04/02/26 23:55:34.116
STEP: creating the HCP cluster @ 04/02/26 23:56:27.658
"ts"="2026-04-02 23:56:27.658606" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cluster-np-labels-taints" "resourceGroup"="rg-np-labels-taints-vvn7hc"
STEP: getting credentials @ 04/03/26 00:05:46.538
STEP: creating the node pool with initial labels and taints @ 04/03/26 00:07:53.808
STEP: waiting for nodes to be ready @ 04/03/26 00:16:31.838
STEP: verifying initial labels are present on nodes @ 04/03/26 00:16:31.986
STEP: verifying initial taints are present on nodes @ 04/03/26 00:16:32.013
STEP: updating nodepool with new taints and scaling up @ 04/03/26 00:16:32.013
STEP: verifying nodes are scaled to the expected count @ 04/03/26 00:29:03.022
STEP: verifying new taint is present on newly created node @ 04/03/26 00:29:03.167
STEP: updating nodepool with a new label and scaling up @ 04/03/26 00:29:03.194
Command Error: signal: killed
Deserializaion Error: no output from commandtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:94]: Unexpected error:
<*fmt.wrapErrors | 0xc000dd4e70>:
failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-2xbb2m" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-2xbb2m, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-2xbb2m\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-2xbb2m, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0010ef9b0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-2xbb2m",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:94]: Unexpected error:
<*fmt.wrapErrors | 0xc000dd4e70>:
failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-2xbb2m" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-2xbb2m, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-2xbb2m\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-2xbb2m, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0010ef9b0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-2xbb2m",
},
<context.deadlineExceededError>{},
],
}
occurredtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
<*fmt.wrapErrors | 0xc0012c8540>:
failed waiting for hcpCluster="admin-cred-lifecycle-t869kp" in resourcegroup="admin-credential-lifecycle-test-qnsckw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-t869kp in resource group admin-credential-lifecycle-test-qnsckw, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-t869kp\" in resourcegroup=\"admin-credential-lifecycle-test-qnsckw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-t869kp in resource group admin-credential-lifecycle-test-qnsckw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00074f9a0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-t869kp in resource group admin-credential-lifecycle-test-qnsckw",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
<*fmt.wrapErrors | 0xc0012c8540>:
failed waiting for hcpCluster="admin-cred-lifecycle-t869kp" in resourcegroup="admin-credential-lifecycle-test-qnsckw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-t869kp in resource group admin-credential-lifecycle-test-qnsckw, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-t869kp\" in resourcegroup=\"admin-credential-lifecycle-test-qnsckw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-t869kp in resource group admin-credential-lifecycle-test-qnsckw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00074f9a0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-t869kp in resource group admin-credential-lifecycle-test-qnsckw",
},
<context.deadlineExceededError>{},
],
}
occurredunknown (1)Process did not finish before 2h0m0s timeout Show raw failure{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-03T00:52:05Z"} |
|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
e2e | failed | n/a | 12 | e2e (12)timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
<*fmt.wrapErrors | 0xc00158e900>:
failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-h9l2tt" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-h9l2tt, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-h9l2tt\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-h9l2tt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0010ff570>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-h9l2tt",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
<*fmt.wrapErrors | 0xc00158e900>:
failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-h9l2tt" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-h9l2tt, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-h9l2tt\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-h9l2tt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0010ff570>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-h9l2tt",
},
<context.deadlineExceededError>{},
],
}
occurredtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error:
<*fmt.wrapErrors | 0xc0017088d0>:
failed waiting for hcpCluster="cluster-stable-4-20-5v4bwn" in resourcegroup="rg-stable-4-20-wttc7h" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-5v4bwn in resource group rg-stable-4-20-wttc7h, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-stable-4-20-5v4bwn\" in resourcegroup=\"rg-stable-4-20-wttc7h\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-5v4bwn in resource group rg-stable-4-20-wttc7h, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ac2e80>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-5v4bwn in resource group rg-stable-4-20-wttc7h",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error:
<*fmt.wrapErrors | 0xc0017088d0>:
failed waiting for hcpCluster="cluster-stable-4-20-5v4bwn" in resourcegroup="rg-stable-4-20-wttc7h" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-5v4bwn in resource group rg-stable-4-20-wttc7h, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-stable-4-20-5v4bwn\" in resourcegroup=\"rg-stable-4-20-wttc7h\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-5v4bwn in resource group rg-stable-4-20-wttc7h, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ac2e80>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-5v4bwn in resource group rg-stable-4-20-wttc7h",
},
<context.deadlineExceededError>{},
],
}
occurredtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:94]: Unexpected error:
<*fmt.wrapErrors | 0xc001606750>:
failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-sv2pc6" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-sv2pc6, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-sv2pc6\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-sv2pc6, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0013a2c10>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-sv2pc6",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:94]: Unexpected error:
<*fmt.wrapErrors | 0xc001606750>:
failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-sv2pc6" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-sv2pc6, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-sv2pc6\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-sv2pc6, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0013a2c10>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-sv2pc6",
},
<context.deadlineExceededError>{},
],
}
occurredtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
<*fmt.wrapErrors | 0xc000669920>:
failed waiting for hcpCluster="cluster-zstream-4-19-qbtwx2" in resourcegroup="rg-zstream-upgrade-4-19-rdbzd7" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-qbtwx2 in resource group rg-zstream-upgrade-4-19-rdbzd7, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-qbtwx2\" in resourcegroup=\"rg-zstream-upgrade-4-19-rdbzd7\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-qbtwx2 in resource group rg-zstream-upgrade-4-19-rdbzd7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000af2380>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-qbtwx2 in resource group rg-zstream-upgrade-4-19-rdbzd7",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
<*fmt.wrapErrors | 0xc000669920>:
failed waiting for hcpCluster="cluster-zstream-4-19-qbtwx2" in resourcegroup="rg-zstream-upgrade-4-19-rdbzd7" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-qbtwx2 in resource group rg-zstream-upgrade-4-19-rdbzd7, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-qbtwx2\" in resourcegroup=\"rg-zstream-upgrade-4-19-rdbzd7\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-qbtwx2 in resource group rg-zstream-upgrade-4-19-rdbzd7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000af2380>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-qbtwx2 in resource group rg-zstream-upgrade-4-19-rdbzd7",
},
<context.deadlineExceededError>{},
],
}
occurredtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
<*fmt.wrapErrors | 0xc001406a20>:
failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-4s2l8t" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4s2l8t, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-4s2l8t\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4s2l8t, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000966f70>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4s2l8t",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
<*fmt.wrapErrors | 0xc001406a20>:
failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-4s2l8t" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4s2l8t, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-4s2l8t\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4s2l8t, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000966f70>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4s2l8t",
},
<context.deadlineExceededError>{},
],
}
occurredtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
<*fmt.wrapErrors | 0xc001786690>:
failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-thfp4w" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-thfp4w, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-thfp4w\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-thfp4w, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001359710>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-thfp4w",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
<*fmt.wrapErrors | 0xc001786690>:
failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-thfp4w" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-thfp4w, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-thfp4w\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-thfp4w, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001359710>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-thfp4w",
},
<context.deadlineExceededError>{},
],
}
occurredInterrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:164]: Unexpected error:
<*fmt.wrapError | 0xc000840ca0>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-sspgrn-nth9sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-sspgrn/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-sspgrn-nth9sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-sspgrn/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001499d70>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["107"],
"Content-Type": ["application/json"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/switzerlandnorth/c0747b09-a0e7-4f8e-8b91-cb4928e98c6c",
],
"X-Ms-Correlation-Request-Id": [
"192eab27-e911-4985-9f2c-186922695500",
],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Failure-Cause": ["service"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: F9256E676FE24D3E97CB455F2F63E23F Ref B: CH1AA2020603045 Ref C: 2026-04-02T22:44:02Z",
],
"Date": [
"Thu, 02 Apr 2026 22:44:02 GMT",
],
"X-Envoy-Upstream-Service-Time": ["127"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Routing-Request-Id": [
"SWITZERLANDNORTH:20260402T224403Z:192eab27-e911-4985-9f2c-186922695500",
],
"X-Content-Type-Options": ["nosniff"],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"d3ed7137-b788-458b-a82f-a51611331a6f",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
},
Body: <*exported.nopClosingBytesReader | 0xc000840be0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
U...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:164]: Unexpected error:
<*fmt.wrapError | 0xc000840ca0>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-sspgrn-nth9sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-sspgrn/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-sspgrn-nth9sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-sspgrn/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001499d70>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["107"],
"Content-Type": ["application/json"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/switzerlandnorth/c0747b09-a0e7-4f8e-8b91-cb4928e98c6c",
],
"X-Ms-Correlation-Request-Id": [
"192eab27-e911-4985-9f2c-186922695500",
],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Failure-Cause": ["service"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: F9256E676FE24D3E97CB455F2F63E23F Ref B: CH1AA2020603045 Ref C: 2026-04-02T22:44:02Z",
],
"Date": [
"Thu, 02 Apr 2026 22:44:02 GMT",
],
"X-Envoy-Upstream-Service-Time": ["127"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Routing-Request-Id": [
"SWITZERLANDNORTH:20260402T224403Z:192eab27-e911-4985-9f2c-186922695500",
],
"X-Content-Type-Options": ["nosniff"],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Request-Id": [
"d3ed7137-b788-458b-a82f-a51611331a6f",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
},
Body: <*exported.nopClosingBytesReader | 0xc000840be0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
U...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfound 1 managed resource groups left behind HCP clusters in cluster-listing-g7w797 Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246]: Unexpected error:
<*errors.joinError | 0xc001010078>:
found 1 managed resource groups left behind HCP clusters in cluster-listing-g7w797
{
errs: [
<*errors.errorString | 0xc00185b800>{
s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-g7w797",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246]: Unexpected error:
<*errors.joinError | 0xc001010078>:
found 1 managed resource groups left behind HCP clusters in cluster-listing-g7w797
{
errs: [
<*errors.errorString | 0xc00185b800>{
s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-g7w797",
},
],
}
occurredDeserializaion Error: no output from command Show raw failure16860, 0xc00182ede0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:297 +0x14d fp=0xc000cedfc0 sp=0xc000cedf28 pc=0x527aeed
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:306 +0x28 fp=0xc000cedfe0 sp=0xc000cedfc0 pc=0x527ad68
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000cedfe8 sp=0xc000cedfe0 pc=0x2dbecc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 793
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:290 +0xc5
goroutine 772 gp=0xc000ecddc0 m=nil [IO wait]:
runtime.gopark(0xc0006328c0?, 0xb0ed2000?, 0x50?, 0x37?, 0xb?)
/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000cf3720 sp=0xc000cf3700 pc=0x2db650e
runtime.netpollblock(0x2e20d18?, 0x2d42066?, 0x0?)
/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc000cf3758 sp=0xc000cf3720 pc=0x2d75fb7
internal/poll.runtime_pollWait(0x7f9fb2f96600, 0x72)
/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc000cf3778 sp=0xc000cf3758 pc=0x2db4da5
internal/poll.(*pollDesc).wait(0xc0008b2d00?, 0xc0011ac000?, 0x0)
/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000cf37a0 sp=0xc000cf3778 pc=0x2e43a47
internal/poll.(*pollDesc).waitRead(...)
/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0008b2d00, {0xc0011ac000, 0x6000, 0x6000})
/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc000cf3838 sp=0xc000cf37a0 pc=0x2e44e79
net.(*netFD).Read(0xc0008b2d00, {0xc0011ac000?, 0xc0011ac2f4?, 0x5?})
/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc000cf3880 sp=0xc000cf3838 pc=0x300a4e5
net.(*conn).Read(0xc0001aa328, {0xc0011ac000?, 0x7f9fb0ed1800?, 0x7f9ff9b7d5c0?})
/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc000cf38c8 sp=0xc000cf3880 pc=0x301c585
crypto/tls.(*atLeastReader).Read(0xc0018c0330, {0xc0011ac000?, 0x5d07?, 0x2db3245?})
/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc000cf3910 sp=0xc000cf38c8 pc=0x308801b
bytes.(*Buffer).ReadFrom(0xc0007c77a8, {0x8c11b40, 0xc0018c0330})
/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc000cf3968 sp=0xc000cf3910 pc=0x2e81c78
crypto/tls.(*Conn).readFromUntil(0xc0007c7508, {0x8c12920, 0xc0001aa328}, 0xc000cf39d0?)
/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc000cf39a0 sp=0xc000cf3968 pc=0x30881fe
crypto/tls.(*Conn).readRecordOrCCS(0xc0007c7508, 0x0)
/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc000cf3c18 sp=0xc000cf39a0 pc=0x308527b
crypto/tls.(*Conn).readRecord(...)
/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc0007c7508, {0xc001131000, 0x1000, 0x34ca980?})
/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc000cf3c80 sp=0xc000cf3c18 pc=0x308bb65
bufio.(*Reader).Read(0xc0010f70e0, {0xc000770824, 0x9, 0x34d8a2e?})
/usr/lib/golang/src/bufio/bufio.go:245 +0x197 fp=0xc000cf3cb8 sp=0xc000cf3c80 pc=0x2f1e1d7
io.ReadAtLeast({0x8c11140, 0xc0010f70e0}, {0xc000770824, 0x9, 0x9}, 0x9)
/usr/lib/golang/src/io/io.go:335 +0x8e fp=0xc000cf3d00 sp=0xc000cf3cb8 pc=0x2e10e8e
io.ReadFull(...)
/usr/lib/golang/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc000770824, 0x9, 0xc000000069?}, {0x8c11140?, 0xc0010f70e0?})
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:250 +0x65 fp=0xc000cf3d50 sp=0xc000cf3d00 pc=0x34c1745
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc0007707e0)
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:513 +0x6b fp=0xc000cf3dd8 sp=0xc000cf3d50 pc=0x34c230b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0007707e0)
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:572 +0x18 fp=0xc000cf3e18 sp=0xc000cf3dd8 pc=0x34c2a78
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000cf3fa8)
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2254 +0xca fp=0xc000cf3f60 sp=0xc000cf3e18 pc=0x34d596a
golang.org/x/net/http2.(*ClientConn).readLoop(0xc0010f8380)
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2123 +0x52 fp=0xc000cf3fc8 sp=0xc000cf3f60 pc=0x34d4cf2
golang.org/x/net/http2.(*Transport).newClientConn.gowrap1()
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0x25 fp=0xc000cf3fe0 sp=0xc000cf3fc8 pc=0x34ce485
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000cf3fe8 sp=0xc000cf3fe0 pc=0x2dbecc1
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 771
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0xda5
rax 0xca
rbx 0x0
rcx 0x2dc0ae3
rdx 0x0
rdi 0xc1ba5b0
rsi 0x80
rbp 0x7ffd3f8d97b8
rsp 0x7ffd3f8d9770
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x6a
r13 0x1
r14 0xc1b84c0
r15 0x5
rip 0x2dc0ae1
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserializaion Error: no output from command
16860, 0xc00182ede0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:297 +0x14d fp=0xc000cedfc0 sp=0xc000cedf28 pc=0x527aeed
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:306 +0x28 fp=0xc000cedfe0 sp=0xc000cedfc0 pc=0x527ad68
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000cedfe8 sp=0xc000cedfe0 pc=0x2dbecc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 793
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:290 +0xc5
goroutine 772 gp=0xc000ecddc0 m=nil [IO wait]:
runtime.gopark(0xc0006328c0?, 0xb0ed2000?, 0x50?, 0x37?, 0xb?)
/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000cf3720 sp=0xc000cf3700 pc=0x2db650e
runtime.netpollblock(0x2e20d18?, 0x2d42066?, 0x0?)
/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc000cf3758 sp=0xc000cf3720 pc=0x2d75fb7
internal/poll.runtime_pollWait(0x7f9fb2f96600, 0x72)
/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc000cf3778 sp=0xc000cf3758 pc=0x2db4da5
internal/poll.(*pollDesc).wait(0xc0008b2d00?, 0xc0011ac000?, 0x0)
/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000cf37a0 sp=0xc000cf3778 pc=0x2e43a47
internal/poll.(*pollDesc).waitRead(...)
/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0008b2d00, {0xc0011ac000, 0x6000, 0x6000})
/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc000cf3838 sp=0xc000cf37a0 pc=0x2e44e79
net.(*netFD).Read(0xc0008b2d00, {0xc0011ac000?, 0xc0011ac2f4?, 0x5?})
/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc000cf3880 sp=0xc000cf3838 pc=0x300a4e5
net.(*conn).Read(0xc0001aa328, {0xc0011ac000?, 0x7f9fb0ed1800?, 0x7f9ff9b7d5c0?})
/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc000cf38c8 sp=0xc000cf3880 pc=0x301c585
crypto/tls.(*atLeastReader).Read(0xc0018c0330, {0xc0011ac000?, 0x5d07?, 0x2db3245?})
/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc000cf3910 sp=0xc000cf38c8 pc=0x308801b
bytes.(*Buffer).ReadFrom(0xc0007c77a8, {0x8c11b40, 0xc0018c0330})
/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc000cf3968 sp=0xc000cf3910 pc=0x2e81c78
crypto/tls.(*Conn).readFromUntil(0xc0007c7508, {0x8c12920, 0xc0001aa328}, 0xc000cf39d0?)
/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc000cf39a0 sp=0xc000cf3968 pc=0x30881fe
crypto/tls.(*Conn).readRecordOrCCS(0xc0007c7508, 0x0)
/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc000cf3c18 sp=0xc000cf39a0 pc=0x308527b
crypto/tls.(*Conn).readRecord(...)
/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc0007c7508, {0xc001131000, 0x1000, 0x34ca980?})
/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc000cf3c80 sp=0xc000cf3c18 pc=0x308bb65
bufio.(*Reader).Read(0xc0010f70e0, {0xc000770824, 0x9, 0x34d8a2e?})
/usr/lib/golang/src/bufio/bufio.go:245 +0x197 fp=0xc000cf3cb8 sp=0xc000cf3c80 pc=0x2f1e1d7
io.ReadAtLeast({0x8c11140, 0xc0010f70e0}, {0xc000770824, 0x9, 0x9}, 0x9)
/usr/lib/golang/src/io/io.go:335 +0x8e fp=0xc000cf3d00 sp=0xc000cf3cb8 pc=0x2e10e8e
io.ReadFull(...)
/usr/lib/golang/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc000770824, 0x9, 0xc000000069?}, {0x8c11140?, 0xc0010f70e0?})
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:250 +0x65 fp=0xc000cf3d50 sp=0xc000cf3d00 pc=0x34c1745
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc0007707e0)
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:513 +0x6b fp=0xc000cf3dd8 sp=0xc000cf3d50 pc=0x34c230b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0007707e0)
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:572 +0x18 fp=0xc000cf3e18 sp=0xc000cf3dd8 pc=0x34c2a78
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000cf3fa8)
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2254 +0xca fp=0xc000cf3f60 sp=0xc000cf3e18 pc=0x34d596a
golang.org/x/net/http2.(*ClientConn).readLoop(0xc0010f8380)
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2123 +0x52 fp=0xc000cf3fc8 sp=0xc000cf3f60 pc=0x34d4cf2
golang.org/x/net/http2.(*Transport).newClientConn.gowrap1()
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0x25 fp=0xc000cf3fe0 sp=0xc000cf3fc8 pc=0x34ce485
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000cf3fe8 sp=0xc000cf3fe0 pc=0x2dbecc1
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 771
/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0xda5
rax 0xca
rbx 0x0
rcx 0x2dc0ae3
rdx 0x0
rdi 0xc1ba5b0
rsi 0x80
rbp 0x7ffd3f8d97b8
rsp 0x7ffd3f8d9770
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x6a
r13 0x1
r14 0xc1b84c0
r15 0x5
rip 0x2dc0ae1
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserializaion Error: no output from commandtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
<*fmt.wrapErrors | 0xc0011e2de0>:
failed waiting for hcpCluster="admin-cred-lifecycle-tmq76j" in resourcegroup="admin-credential-lifecycle-test-fvb2sn" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-tmq76j in resource group admin-credential-lifecycle-test-fvb2sn, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-tmq76j\" in resourcegroup=\"admin-credential-lifecycle-test-fvb2sn\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-tmq76j in resource group admin-credential-lifecycle-test-fvb2sn, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001575f50>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-tmq76j in resource group admin-credential-lifecycle-test-fvb2sn",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
<*fmt.wrapErrors | 0xc0011e2de0>:
failed waiting for hcpCluster="admin-cred-lifecycle-tmq76j" in resourcegroup="admin-credential-lifecycle-test-fvb2sn" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-tmq76j in resource group admin-credential-lifecycle-test-fvb2sn, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-tmq76j\" in resourcegroup=\"admin-credential-lifecycle-test-fvb2sn\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-tmq76j in resource group admin-credential-lifecycle-test-fvb2sn, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001575f50>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-tmq76j in resource group admin-credential-lifecycle-test-fvb2sn",
},
<context.deadlineExceededError>{},
],
}
occurredtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
<*fmt.wrapErrors | 0xc000e80390>:
failed waiting for hcpCluster="cluster-zstream-4-21-dh4k5t" in resourcegroup="rg-zstream-upgrade-4-21-5xwxz7" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-dh4k5t in resource group rg-zstream-upgrade-4-21-5xwxz7, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-dh4k5t\" in resourcegroup=\"rg-zstream-upgrade-4-21-5xwxz7\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-dh4k5t in resource group rg-zstream-upgrade-4-21-5xwxz7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0006a10b0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-dh4k5t in resource group rg-zstream-upgrade-4-21-5xwxz7",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
<*fmt.wrapErrors | 0xc000e80390>:
failed waiting for hcpCluster="cluster-zstream-4-21-dh4k5t" in resourcegroup="rg-zstream-upgrade-4-21-5xwxz7" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-dh4k5t in resource group rg-zstream-upgrade-4-21-5xwxz7, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-dh4k5t\" in resourcegroup=\"rg-zstream-upgrade-4-21-5xwxz7\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-dh4k5t in resource group rg-zstream-upgrade-4-21-5xwxz7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0006a10b0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-dh4k5t in resource group rg-zstream-upgrade-4-21-5xwxz7",
},
<context.deadlineExceededError>{},
],
}
occurred |
|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
e2e | failed | n/a | 9 | e2e (9)timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
<*fmt.wrapErrors | 0xc000dbeed0>:
failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-4v6pcw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4v6pcw, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-4v6pcw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4v6pcw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000a57300>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4v6pcw",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
<*fmt.wrapErrors | 0xc000dbeed0>:
failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-4v6pcw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4v6pcw, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-4v6pcw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4v6pcw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000a57300>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4v6pcw",
},
<context.deadlineExceededError>{},
],
}
occurredERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
<*fmt.wrapError | 0xc0010bac80>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-hzrfhf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-hzrfhf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001203e30>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["107"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Correlation-Request-Id": [
"e1ea6c99-e1af-4578-aef0-1306b523c906",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"X-Ms-Request-Id": [
"2d2345a2-c74d-4312-a848-a40bf84bd584",
],
"X-Ms-Failure-Cause": ["service"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Expires": ["-1"],
"X-Envoy-Upstream-Service-Time": ["841"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/7287e47d-3f8a-4d8b-8e48-1036f8537975",
],
"X-Msedge-Ref": [
"Ref A: 99D6B712DDED4198801BF5D06DC67B75 Ref B: CHI261140501040 Ref C: 2026-04-02T21:15:54Z",
],
"Date": [
"Thu, 02 Apr 2026 21:15:54 GMT",
],
"Pragma": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260402T211555Z:e1ea6c99-e1af-4578-aef0-1306b523c906",
],
},
Body: <*exported.nopClosingBytesReader | 0xc0010babc0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
<*fmt.wrapError | 0xc0010bac80>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-hzrfhf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-hzrfhf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001203e30>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"Content-Length": ["107"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"X-Ms-Correlation-Request-Id": [
"e1ea6c99-e1af-4578-aef0-1306b523c906",
],
"X-Content-Type-Options": ["nosniff"],
"X-Cache": ["CONFIG_NOCACHE"],
"Cache-Control": ["no-cache"],
"X-Ms-Request-Id": [
"2d2345a2-c74d-4312-a848-a40bf84bd584",
],
"X-Ms-Failure-Cause": ["service"],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Expires": ["-1"],
"X-Envoy-Upstream-Service-Time": ["841"],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/7287e47d-3f8a-4d8b-8e48-1036f8537975",
],
"X-Msedge-Ref": [
"Ref A: 99D6B712DDED4198801BF5D06DC67B75 Ref B: CHI261140501040 Ref C: 2026-04-02T21:15:54Z",
],
"Date": [
"Thu, 02 Apr 2026 21:15:54 GMT",
],
"Pragma": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260402T211555Z:e1ea6c99-e1af-4578-aef0-1306b523c906",
],
},
Body: <*exported.nopClosingBytesReader | 0xc0010babc0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredInterrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:172]: Unexpected error:
<*fmt.wrapError | 0xc000a56060>:
failed creating external auth "external-auth" in resourcegroup="external-auth-cluster-vvkmt7" for cluster="ea-cluster": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-vvkmt7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed creating external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-vvkmt7\" for cluster=\"ea-cluster\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-vvkmt7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000959e60>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Envoy-Upstream-Service-Time": ["602"],
"X-Ms-Correlation-Request-Id": [
"3bba6e4b-62fc-4679-ac86-6cca778ca60d",
],
"X-Msedge-Ref": [
"Ref A: E927148DA18943C493DC7980C427DBBF Ref B: CHI261140511042 Ref C: 2026-04-02T21:28:00Z",
],
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"c4a7fd9a-75bf-42b2-9162-e851d965413a",
],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/6c5a5ff0-966d-49cb-9120-62d22bd412e9",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260402T212801Z:3bba6e4b-62fc-4679-ac86-6cca778ca60d",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"Date": [
"Thu, 02 Apr 2026 21:28:00 GMT",
],
"Cache-Control": ["no-cache"],
"Content-Length": ["107"],
"X-Cache": ["CONFIG_NOCACHE"],
},
Body: <*exported.nopClosingBytesReader | 0xc000b1df20>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:172]: Unexpected error:
<*fmt.wrapError | 0xc000a56060>:
failed creating external auth "external-auth" in resourcegroup="external-auth-cluster-vvkmt7" for cluster="ea-cluster": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-vvkmt7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed creating external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-vvkmt7\" for cluster=\"ea-cluster\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-vvkmt7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000959e60>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Envoy-Upstream-Service-Time": ["602"],
"X-Ms-Correlation-Request-Id": [
"3bba6e4b-62fc-4679-ac86-6cca778ca60d",
],
"X-Msedge-Ref": [
"Ref A: E927148DA18943C493DC7980C427DBBF Ref B: CHI261140511042 Ref C: 2026-04-02T21:28:00Z",
],
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"c4a7fd9a-75bf-42b2-9162-e851d965413a",
],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Pragma": ["no-cache"],
"Expires": ["-1"],
"X-Ms-Error-Code": [
"InternalServerError",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/6c5a5ff0-966d-49cb-9120-62d22bd412e9",
],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260402T212801Z:3bba6e4b-62fc-4679-ac86-6cca778ca60d",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"X-Content-Type-Options": ["nosniff"],
"Date": [
"Thu, 02 Apr 2026 21:28:00 GMT",
],
"Cache-Control": ["no-cache"],
"Content-Length": ["107"],
"X-Cache": ["CONFIG_NOCACHE"],
},
Body: <*exported.nopClosingBytesReader | 0xc000b1df20>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PUT",
URL: {
Scheme: "https",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
<*fmt.wrapErrors | 0xc0015b4ff0>:
failed waiting for hcpCluster="cluster-zstream-4-20-cjp2ct" in resourcegroup="rg-zstream-upgrade-4-20-92d5lx" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-cjp2ct in resource group rg-zstream-upgrade-4-20-92d5lx, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-cjp2ct\" in resourcegroup=\"rg-zstream-upgrade-4-20-92d5lx\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-cjp2ct in resource group rg-zstream-upgrade-4-20-92d5lx, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001687e10>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-cjp2ct in resource group rg-zstream-upgrade-4-20-92d5lx",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
<*fmt.wrapErrors | 0xc0015b4ff0>:
failed waiting for hcpCluster="cluster-zstream-4-20-cjp2ct" in resourcegroup="rg-zstream-upgrade-4-20-92d5lx" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-cjp2ct in resource group rg-zstream-upgrade-4-20-92d5lx, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-cjp2ct\" in resourcegroup=\"rg-zstream-upgrade-4-20-92d5lx\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-cjp2ct in resource group rg-zstream-upgrade-4-20-92d5lx, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001687e10>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-cjp2ct in resource group rg-zstream-upgrade-4-20-92d5lx",
},
<context.deadlineExceededError>{},
],
}
occurredtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:94]: Unexpected error:
<*fmt.wrapErrors | 0xc000f5aa80>:
failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-xn9npb" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-xn9npb, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-xn9npb\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-xn9npb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0008e33b0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-xn9npb",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:94]: Unexpected error:
<*fmt.wrapErrors | 0xc000f5aa80>:
failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-xn9npb" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-xn9npb, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-xn9npb\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-xn9npb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0008e33b0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-xn9npb",
},
<context.deadlineExceededError>{},
],
}
occurredERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error:
<*fmt.wrapError | 0xc00075a160>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-stable-4-20-zchh4x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-stable-4-20-gcjhh8/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-stable-4-20-zchh4x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-stable-4-20-gcjhh8/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001206570>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Content-Type-Options": ["nosniff"],
"Content-Length": ["107"],
"Expires": ["-1"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260402T214410Z:4a38c176-8867-44b2-a3c7-48492a8d4e0f",
],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: 79D0092FC1DF43CBAC048BA3F59FDC4E Ref B: CHI261140505040 Ref C: 2026-04-02T21:44:10Z",
],
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"cb3209b7-3062-4e31-95f1-095a20aeff07",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/c6802a55-7b52-4d68-ac18-173dfd506998",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Date": [
"Thu, 02 Apr 2026 21:44:10 GMT",
],
"X-Ms-Error-Code": [
"InternalServerError",
],
"Pragma": ["no-cache"],
"X-Envoy-Upstream-Service-Time": ["219"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Correlation-Request-Id": [
"4a38c176-8867-44b2-a3c7-48492a8d4e0f",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
},
Body: <*exported.nopClosingBytesReader | 0xc00075a060>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error:
<*fmt.wrapError | 0xc00075a160>:
failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-stable-4-20-zchh4x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-stable-4-20-gcjhh8/requestAdminCredential
--------------------------------------------------------------------------------
RESPONSE 500: 500 Internal Server Error
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"error": {
"code": "InternalServerError",
"message": "Internal server error."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-stable-4-20-zchh4x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-stable-4-20-gcjhh8/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001206570>{
ErrorCode: "InternalServerError",
StatusCode: 500,
RawResponse: {
Status: "500 Internal Server Error",
StatusCode: 500,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Content-Type-Options": ["nosniff"],
"Content-Length": ["107"],
"Expires": ["-1"],
"X-Ms-Failure-Cause": ["service"],
"X-Ms-Routing-Request-Id": [
"CANADACENTRAL:20260402T214410Z:4a38c176-8867-44b2-a3c7-48492a8d4e0f",
],
"X-Cache": ["CONFIG_NOCACHE"],
"X-Msedge-Ref": [
"Ref A: 79D0092FC1DF43CBAC048BA3F59FDC4E Ref B: CHI261140505040 Ref C: 2026-04-02T21:44:10Z",
],
"Cache-Control": ["no-cache"],
"Content-Type": ["application/json"],
"X-Ms-Request-Id": [
"cb3209b7-3062-4e31-95f1-095a20aeff07",
],
"X-Ms-Operation-Identifier": [
"appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/c6802a55-7b52-4d68-ac18-173dfd506998",
],
"X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
"Date": [
"Thu, 02 Apr 2026 21:44:10 GMT",
],
"X-Ms-Error-Code": [
"InternalServerError",
],
"Pragma": ["no-cache"],
"X-Envoy-Upstream-Service-Time": ["219"],
"X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
"X-Ms-Correlation-Request-Id": [
"4a38c176-8867-44b2-a3c7-48492a8d4e0f",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
},
Body: <*exported.nopClosingBytesReader | 0xc00075a060>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 107,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "POST",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
<*fmt.wrapErrors | 0xc001568360>:
failed waiting for hcpCluster="cluster-zstream-4-19-45hj5h" in resourcegroup="rg-zstream-upgrade-4-19-g7skvb" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-45hj5h in resource group rg-zstream-upgrade-4-19-g7skvb, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-45hj5h\" in resourcegroup=\"rg-zstream-upgrade-4-19-g7skvb\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-45hj5h in resource group rg-zstream-upgrade-4-19-g7skvb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0010628b0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-45hj5h in resource group rg-zstream-upgrade-4-19-g7skvb",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
<*fmt.wrapErrors | 0xc001568360>:
failed waiting for hcpCluster="cluster-zstream-4-19-45hj5h" in resourcegroup="rg-zstream-upgrade-4-19-g7skvb" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-45hj5h in resource group rg-zstream-upgrade-4-19-g7skvb, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-45hj5h\" in resourcegroup=\"rg-zstream-upgrade-4-19-g7skvb\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-45hj5h in resource group rg-zstream-upgrade-4-19-g7skvb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0010628b0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-45hj5h in resource group rg-zstream-upgrade-4-19-g7skvb",
},
<context.deadlineExceededError>{},
],
}
occurredtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
<*fmt.wrapErrors | 0xc000c1a9f0>:
failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-kldf9f" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-kldf9f, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-kldf9f\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-kldf9f, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0015c97d0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-kldf9f",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
<*fmt.wrapErrors | 0xc000c1a9f0>:
failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-kldf9f" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-kldf9f, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-kldf9f\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-kldf9f, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0015c97d0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-kldf9f",
},
<context.deadlineExceededError>{},
],
}
occurred |
|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
e2e | failed | n/a | 5 | e2e (5)Deserializaion Error: no output from command Show raw failure/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:56 +0x1e fp=0xc00173bb78 sp=0xc00173bb58 pc=0x3af095e
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters({0x7f6251aa1260, 0xc000c5c210}, 0xc000de52a8, {0xc000d81c20, 0x1e}, 0x34630b8a000)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:352 +0x8ef fp=0xc00173be20 sp=0xc00173bb78 pc=0x525e40f
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).cleanupResourceGroup(0xc0009a4780, {0x7f6251aa1260, 0xc000c5c210}, {0xc000d81c20, 0x1e}, 0x34630b8a000)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:436 +0x294 fp=0xc00173bf28 sp=0xc00173be20 pc=0x527c3b4
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7f6251aa1260, 0xc000c5c210})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:297 +0x14d fp=0xc00173bfc0 sp=0xc00173bf28 pc=0x527aeed
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:306 +0x28 fp=0xc00173bfe0 sp=0xc00173bfc0 pc=0x527ad68
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00173bfe8 sp=0xc00173bfe0 pc=0x2dbecc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 589
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:290 +0xc5
goroutine 589 gp=0xc001544e00 m=nil [sync.WaitGroup.Wait, 5 minutes]:
runtime.gopark(0xc000da95f0?, 0x2d4d874?, 0x60?, 0x6f?, 0x200c0016309a0?)
/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000c65590 sp=0xc000c65570 pc=0x2db650e
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:466
runtime.semacquire1(0xc00118f8c8, 0x0, 0x1, 0x0, 0x19)
/usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc000c655f8 sp=0xc000c65590 pc=0x2d9259d
sync.runtime_SemacquireWaitGroup(0xc0007953a0?, 0x0?)
/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc000c65630 sp=0xc000c655f8 pc=0x2db7c2e
sync.(*WaitGroup).Wait(0xc00118f8c0)
/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc000c65658 sp=0xc000c65630 pc=0x2ddb1a5
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups(0xc0009a4780, {0x7f6251aa1260, 0xc000c5c210}, {{0xc00097ad80, 0x1, 0x1}, 0x34630b8a000, {0x81c7dce, 0x8}})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:308 +0x273 fp=0xc000c65768 sp=0xc000c65658 pc=0x527a9b3
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources(0xc0009a4780, {0x7f6251aa1260, 0xc000c5c210})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:227 +0x225 fp=0xc000c65888 sp=0xc000c65768 pc=0x527a245
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources-fm({0x7f6251aa1260?, 0xc000c5c210?})
<autogenerated>:1 +0x33 fp=0xc000c658b0 sp=0xc000c65888 pc=0x528ba73
runtime.call16(0xc00147f710, 0xc000b04760, 0x0, 0x0, 0x0, 0x10, 0xc000c65b08)
/usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc000c658d0 sp=0xc000c658b0 pc=0x2dbd223
runtime.reflectcall(0x7bc5d80?, 0xc000c5c210?, 0x2?, 0x81fbd86?, 0x0?, 0x12?, 0x77c4e60?)
<autogenerated>:1 +0x36 fp=0xc000c65910 sp=0xc000c658d0 pc=0x2dc5ab6
reflect.Value.call({0x70d3280?, 0xc000b04760?, 0xc1b6f40?}, {0x81bbe44, 0x4}, {0xc00010df60, 0x1, 0xc000b98388?})
/usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc000c65e70 sp=0xc000c65910 pc=0x2df8a86
reflect.Value.Call({0x70d3280?, 0xc000b04760?, 0x0?}, {0xc00010df60?, 0xc00010de50?, 0xc00010df68?})
/usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc000c65ee8 sp=0xc000c65e70 pc=0x2df7b79
github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x8cc8ce0?, 0xc000c5c210?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc000c65f88 sp=0xc000c65ee8 pc=0x321aff0
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000c65fe0 sp=0xc000c65f88 pc=0x322dabb
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000c65fe8 sp=0xc000c65fe0 pc=0x2dbecc1
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 86
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50
rax 0xca
rbx 0x0
rcx 0x2dc0ae3
rdx 0x0
rdi 0xc1ba5b0
rsi 0x80
rbp 0x7ffe92ed0ff0
rsp 0x7ffe92ed0fa8
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x0
r13 0x1
r14 0xc1b84c0
r15 0x7
rip 0x2dc0ae1
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserializaion Error: no output from command
/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:56 +0x1e fp=0xc00173bb78 sp=0xc00173bb58 pc=0x3af095e
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters({0x7f6251aa1260, 0xc000c5c210}, 0xc000de52a8, {0xc000d81c20, 0x1e}, 0x34630b8a000)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:352 +0x8ef fp=0xc00173be20 sp=0xc00173bb78 pc=0x525e40f
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).cleanupResourceGroup(0xc0009a4780, {0x7f6251aa1260, 0xc000c5c210}, {0xc000d81c20, 0x1e}, 0x34630b8a000)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:436 +0x294 fp=0xc00173bf28 sp=0xc00173be20 pc=0x527c3b4
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7f6251aa1260, 0xc000c5c210})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:297 +0x14d fp=0xc00173bfc0 sp=0xc00173bf28 pc=0x527aeed
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:306 +0x28 fp=0xc00173bfe0 sp=0xc00173bfc0 pc=0x527ad68
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00173bfe8 sp=0xc00173bfe0 pc=0x2dbecc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 589
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:290 +0xc5
goroutine 589 gp=0xc001544e00 m=nil [sync.WaitGroup.Wait, 5 minutes]:
runtime.gopark(0xc000da95f0?, 0x2d4d874?, 0x60?, 0x6f?, 0x200c0016309a0?)
/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000c65590 sp=0xc000c65570 pc=0x2db650e
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:466
runtime.semacquire1(0xc00118f8c8, 0x0, 0x1, 0x0, 0x19)
/usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc000c655f8 sp=0xc000c65590 pc=0x2d9259d
sync.runtime_SemacquireWaitGroup(0xc0007953a0?, 0x0?)
/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc000c65630 sp=0xc000c655f8 pc=0x2db7c2e
sync.(*WaitGroup).Wait(0xc00118f8c0)
/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc000c65658 sp=0xc000c65630 pc=0x2ddb1a5
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups(0xc0009a4780, {0x7f6251aa1260, 0xc000c5c210}, {{0xc00097ad80, 0x1, 0x1}, 0x34630b8a000, {0x81c7dce, 0x8}})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:308 +0x273 fp=0xc000c65768 sp=0xc000c65658 pc=0x527a9b3
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources(0xc0009a4780, {0x7f6251aa1260, 0xc000c5c210})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:227 +0x225 fp=0xc000c65888 sp=0xc000c65768 pc=0x527a245
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources-fm({0x7f6251aa1260?, 0xc000c5c210?})
<autogenerated>:1 +0x33 fp=0xc000c658b0 sp=0xc000c65888 pc=0x528ba73
runtime.call16(0xc00147f710, 0xc000b04760, 0x0, 0x0, 0x0, 0x10, 0xc000c65b08)
/usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc000c658d0 sp=0xc000c658b0 pc=0x2dbd223
runtime.reflectcall(0x7bc5d80?, 0xc000c5c210?, 0x2?, 0x81fbd86?, 0x0?, 0x12?, 0x77c4e60?)
<autogenerated>:1 +0x36 fp=0xc000c65910 sp=0xc000c658d0 pc=0x2dc5ab6
reflect.Value.call({0x70d3280?, 0xc000b04760?, 0xc1b6f40?}, {0x81bbe44, 0x4}, {0xc00010df60, 0x1, 0xc000b98388?})
/usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc000c65e70 sp=0xc000c65910 pc=0x2df8a86
reflect.Value.Call({0x70d3280?, 0xc000b04760?, 0x0?}, {0xc00010df60?, 0xc00010de50?, 0xc00010df68?})
/usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc000c65ee8 sp=0xc000c65e70 pc=0x2df7b79
github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x8cc8ce0?, 0xc000c5c210?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc000c65f88 sp=0xc000c65ee8 pc=0x321aff0
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000c65fe0 sp=0xc000c65f88 pc=0x322dabb
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000c65fe8 sp=0xc000c65fe0 pc=0x2dbecc1
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 86
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50
rax 0xca
rbx 0x0
rcx 0x2dc0ae3
rdx 0x0
rdi 0xc1ba5b0
rsi 0x80
rbp 0x7ffe92ed0ff0
rsp 0x7ffe92ed0fa8
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x0
r13 0x1
r14 0xc1b84c0
r15 0x7
rip 0x2dc0ae1
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserializaion Error: no output from commandtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:159]: Unexpected error:
<*fmt.wrapErrors | 0xc001359200>:
failed waiting for hcpCluster="np-version-upgrade-cluster-6zntpn" in resourcegroup="rg-np-version-upgrade-6zntpn-9zhbdp" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-6zntpn in resource group rg-np-version-upgrade-6zntpn-9zhbdp, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"np-version-upgrade-cluster-6zntpn\" in resourcegroup=\"rg-np-version-upgrade-6zntpn-9zhbdp\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-6zntpn in resource group rg-np-version-upgrade-6zntpn-9zhbdp, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000927ca0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-6zntpn in resource group rg-np-version-upgrade-6zntpn-9zhbdp",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:159]: Unexpected error:
<*fmt.wrapErrors | 0xc001359200>:
failed waiting for hcpCluster="np-version-upgrade-cluster-6zntpn" in resourcegroup="rg-np-version-upgrade-6zntpn-9zhbdp" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-6zntpn in resource group rg-np-version-upgrade-6zntpn-9zhbdp, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"np-version-upgrade-cluster-6zntpn\" in resourcegroup=\"rg-np-version-upgrade-6zntpn-9zhbdp\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-6zntpn in resource group rg-np-version-upgrade-6zntpn-9zhbdp, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000927ca0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-6zntpn in resource group rg-np-version-upgrade-6zntpn-9zhbdp",
},
<context.deadlineExceededError>{},
],
}
occurredtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error:
<*fmt.wrapErrors | 0xc0008d4e70>:
failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-8qgh84" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-8qgh84, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-8qgh84\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-8qgh84, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0000d5cc0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-8qgh84",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error:
<*fmt.wrapErrors | 0xc0008d4e70>:
failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-8qgh84" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-8qgh84, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-8qgh84\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-8qgh84, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0000d5cc0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-8qgh84",
},
<context.deadlineExceededError>{},
],
}
occurredtimeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:159]: Unexpected error:
<*fmt.wrapErrors | 0xc0014890b0>:
failed waiting for hcpCluster="np-version-upgrade-cluster-99z6nv" in resourcegroup="rg-np-version-upgrade-99z6nv-2gt94h" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-99z6nv in resource group rg-np-version-upgrade-99z6nv-2gt94h, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"np-version-upgrade-cluster-99z6nv\" in resourcegroup=\"rg-np-version-upgrade-99z6nv-2gt94h\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-99z6nv in resource group rg-np-version-upgrade-99z6nv-2gt94h, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0014706c0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-99z6nv in resource group rg-np-version-upgrade-99z6nv-2gt94h",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:159]: Unexpected error:
<*fmt.wrapErrors | 0xc0014890b0>:
failed waiting for hcpCluster="np-version-upgrade-cluster-99z6nv" in resourcegroup="rg-np-version-upgrade-99z6nv-2gt94h" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-99z6nv in resource group rg-np-version-upgrade-99z6nv-2gt94h, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"np-version-upgrade-cluster-99z6nv\" in resourcegroup=\"rg-np-version-upgrade-99z6nv-2gt94h\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-99z6nv in resource group rg-np-version-upgrade-99z6nv-2gt94h, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0014706c0>{
s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-99z6nv in resource group rg-np-version-upgrade-99z6nv-2gt94h",
},
<context.deadlineExceededError>{},
],
}
occurredDeserializaion Error: no output from command Show raw failurerupted by User
First interrupt received; Ginkgo will run any cleanup and reporting nodes but will skip all remaining specs. Interrupt again to skip cleanup.
Here's a current progress report:
Customer should be able to create an HCP cluster and custom node pool osDisk size (Spec Runtime: 1h29m59.811s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:35
In [It] (Node Runtime: 1h29m59.811s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:35
At [By Step] getting credentials (Step Runtime: 3m7.813s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96
Spec Goroutine
goroutine 132 [select]
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8cb0998, 0xc000530fc0}, 0x81dda58?)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8cfd5c0, {0x8cb0998, 0xc000530fc0}, 0xc000d10af0)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetAdminRESTConfigForHCPCluster(0xc0007f3cc0, {0x7f25e426ce60, 0xc0009475f0}, 0xc000df1f30, {0xc0016121f8, 0x13}, {0x81fbb34, 0x12}, 0x8bb2c97000)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:107
> github.com/Azure/ARO-HCP/test/e2e.init.func9.2({0x7f25e426ce60, 0xc0009475f0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:97
|
| By("getting credentials")
> adminRESTConfig, err := tc.GetAdminRESTConfigForHCPCluster(
| ctx,
| tc.Get20240610ClientFactoryOrDie(ctx).NewHcpOpenShiftClustersClient(),
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8cc8ce0?, 0xc0009475f0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 129
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104 @ 04/02/26 20:55:25.21
"ts"="2026-04-02 20:55:25.211124" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 20:55:25.211230" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-hcp-cluster-np-128-pjn74q" "resourceGroup"="clusternp128-7zlf92"
"ts"="2026-04-02 20:55:25.489641" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-hcp-cluster-np-128-pjn74q" "resourceGroup"=""
"ts"="2026-04-02 20:55:28.102668" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-a85d76e69c62fcc4973899bcc5957cf5e02900c7383d8dbab637c1fde0fca561.yaml.gz"
"ts"="2026-04-02 20:55:28.102788" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-a85d76e69c62fcc4973899bcc5957cf5e02900c7383d8dbab637c1fde0fca561.yaml.gz"
"ts"="2026-04-02 20:55:28.102850" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 20:55:28.104321" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-78"
"ts"="2026-04-02 20:55:28.104333" "level"=0 "msg"="collecting deployments" "resourceGroup"="clusternp128-7zlf92"
"ts"="2026-04-02 20:55:28.256263" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-78" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 20:55:28.426031" "level"=0 "msg"="collecting operations" "resourceGroup"="clusternp128-7zlf92" "deployment"="pooledNonMsiScopedAssignments-3ljyjjx2n7gxo"
"ts"="2026-04-02 20:55:28.560024" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-78" "deployment"="msi-deployment-78"
"ts"="2026-04-02 20:55:28.696547" "level"=0 "msg"="collecting operations" "resourceGroup"="clusternp128-7zlf92" "deployment"="customer-infra-hcp-cluster-np-128-pjn74q"
"ts"="2026-04-02 20:55:28.998126" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 20:55:28.998270" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 20:55:28.998395" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="clusternp128-7zlf92"
Command Error: signal: killed
Deserializaion Error: no output from command
rupted by User
First interrupt received; Ginkgo will run any cleanup and reporting nodes but will skip all remaining specs. Interrupt again to skip cleanup.
Here's a current progress report:
Customer should be able to create an HCP cluster and custom node pool osDisk size (Spec Runtime: 1h29m59.811s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:35
In [It] (Node Runtime: 1h29m59.811s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:35
At [By Step] getting credentials (Step Runtime: 3m7.813s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96
Spec Goroutine
goroutine 132 [select]
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8cb0998, 0xc000530fc0}, 0x81dda58?)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8cfd5c0, {0x8cb0998, 0xc000530fc0}, 0xc000d10af0)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetAdminRESTConfigForHCPCluster(0xc0007f3cc0, {0x7f25e426ce60, 0xc0009475f0}, 0xc000df1f30, {0xc0016121f8, 0x13}, {0x81fbb34, 0x12}, 0x8bb2c97000)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:107
> github.com/Azure/ARO-HCP/test/e2e.init.func9.2({0x7f25e426ce60, 0xc0009475f0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:97
|
| By("getting credentials")
> adminRESTConfig, err := tc.GetAdminRESTConfigForHCPCluster(
| ctx,
| tc.Get20240610ClientFactoryOrDie(ctx).NewHcpOpenShiftClustersClient(),
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8cc8ce0?, 0xc0009475f0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 129
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104 @ 04/02/26 20:55:25.21
"ts"="2026-04-02 20:55:25.211124" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 20:55:25.211230" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-hcp-cluster-np-128-pjn74q" "resourceGroup"="clusternp128-7zlf92"
"ts"="2026-04-02 20:55:25.489641" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-hcp-cluster-np-128-pjn74q" "resourceGroup"=""
"ts"="2026-04-02 20:55:28.102668" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-a85d76e69c62fcc4973899bcc5957cf5e02900c7383d8dbab637c1fde0fca561.yaml.gz"
"ts"="2026-04-02 20:55:28.102788" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-a85d76e69c62fcc4973899bcc5957cf5e02900c7383d8dbab637c1fde0fca561.yaml.gz"
"ts"="2026-04-02 20:55:28.102850" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 20:55:28.104321" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-78"
"ts"="2026-04-02 20:55:28.104333" "level"=0 "msg"="collecting deployments" "resourceGroup"="clusternp128-7zlf92"
"ts"="2026-04-02 20:55:28.256263" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-78" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 20:55:28.426031" "level"=0 "msg"="collecting operations" "resourceGroup"="clusternp128-7zlf92" "deployment"="pooledNonMsiScopedAssignments-3ljyjjx2n7gxo"
"ts"="2026-04-02 20:55:28.560024" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-78" "deployment"="msi-deployment-78"
"ts"="2026-04-02 20:55:28.696547" "level"=0 "msg"="collecting operations" "resourceGroup"="clusternp128-7zlf92" "deployment"="customer-infra-hcp-cluster-np-128-pjn74q"
"ts"="2026-04-02 20:55:28.998126" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 20:55:28.998270" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 20:55:28.998395" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="clusternp128-7zlf92"
Command Error: signal: killed
Deserializaion Error: no output from command |
|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
e2e | failed | n/a | 2 | e2e (2)Interrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User Interrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User |
|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
e2e | failed | n/a | 4 | e2e (4)Interrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User Deserializaion Error: no output from command Show raw failureting nodes but will skip all remaining specs. Interrupt again to skip cleanup.
Here's a current progress report:
Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API (Spec Runtime: 1h29m59.877s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:42
In [It] (Node Runtime: 1h29m59.877s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:42
At [By Step] getting admin credentials for the cluster (Step Runtime: 5m38.233s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:157
Spec Goroutine
goroutine 87 [select]
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8cb0998, 0xc000a76850}, 0x81dda58?)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8cfd5c0, {0x8cb0998, 0xc000a76850}, 0xc000a449f0)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetAdminRESTConfigForHCPCluster(0xc000b14640, {0x7fa0219e8860, 0xc000c710e0}, 0xc0000d9ec8, {0xc001810108, 0x17}, {0x81fbb46, 0x12}, 0x8bb2c97000)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:107
> github.com/Azure/ARO-HCP/test/e2e.init.func10.2({0x7fa0219e8860, 0xc000c710e0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:158
|
| By("getting admin credentials for the cluster")
> adminRESTConfig, err := tc.GetAdminRESTConfigForHCPCluster(
| ctx,
| tc.Get20240610ClientFactoryOrDie(ctx).NewHcpOpenShiftClustersClient(),
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8cc8ce0?, 0xc000c710e0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 40
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:165 @ 04/02/26 19:42:28.688
"ts"="2026-04-02 19:42:28.689138" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 19:42:28.689210" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-private-kv-cluster-brmrkf" "resourceGroup"="private-keyvault-w6qm7t"
"ts"="2026-04-02 19:42:28.802742" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-private-kv-cluster-brmrkf" "resourceGroup"=""
"ts"="2026-04-02 19:42:29.228630" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-65e261cf588b9f2637c69902ef65386ae7e651e35485a39612d668d6a3309cde.yaml.gz"
"ts"="2026-04-02 19:42:29.228771" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-65e261cf588b9f2637c69902ef65386ae7e651e35485a39612d668d6a3309cde.yaml.gz"
"ts"="2026-04-02 19:42:29.228877" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 19:42:29.231553" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-18"
"ts"="2026-04-02 19:42:29.231567" "level"=0 "msg"="collecting deployments" "resourceGroup"="private-keyvault-w6qm7t"
"ts"="2026-04-02 19:42:29.323497" "level"=0 "msg"="collecting operations" "resourceGroup"="private-keyvault-w6qm7t" "deployment"="pooledNonMsiScopedAssignments-had2isw6vkgre"
"ts"="2026-04-02 19:42:29.375537" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-18" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 19:42:29.376346" "level"=0 "msg"="collecting operations" "resourceGroup"="private-keyvault-w6qm7t" "deployment"="customer-infra-private-kv-cluster-brmrkf"
"ts"="2026-04-02 19:42:29.521817" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-18" "deployment"="msi-deployment-18"
"ts"="2026-04-02 19:42:29.680009" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 19:42:29.680123" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 19:42:29.680252" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="private-keyvault-w6qm7t"
Command Error: signal: killed
Deserializaion Error: no output from command
ting nodes but will skip all remaining specs. Interrupt again to skip cleanup.
Here's a current progress report:
Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API (Spec Runtime: 1h29m59.877s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:42
In [It] (Node Runtime: 1h29m59.877s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:42
At [By Step] getting admin credentials for the cluster (Step Runtime: 5m38.233s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:157
Spec Goroutine
goroutine 87 [select]
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8cb0998, 0xc000a76850}, 0x81dda58?)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8cfd5c0, {0x8cb0998, 0xc000a76850}, 0xc000a449f0)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetAdminRESTConfigForHCPCluster(0xc000b14640, {0x7fa0219e8860, 0xc000c710e0}, 0xc0000d9ec8, {0xc001810108, 0x17}, {0x81fbb46, 0x12}, 0x8bb2c97000)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:107
> github.com/Azure/ARO-HCP/test/e2e.init.func10.2({0x7fa0219e8860, 0xc000c710e0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:158
|
| By("getting admin credentials for the cluster")
> adminRESTConfig, err := tc.GetAdminRESTConfigForHCPCluster(
| ctx,
| tc.Get20240610ClientFactoryOrDie(ctx).NewHcpOpenShiftClustersClient(),
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8cc8ce0?, 0xc000c710e0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 40
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:165 @ 04/02/26 19:42:28.688
"ts"="2026-04-02 19:42:28.689138" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 19:42:28.689210" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-private-kv-cluster-brmrkf" "resourceGroup"="private-keyvault-w6qm7t"
"ts"="2026-04-02 19:42:28.802742" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-private-kv-cluster-brmrkf" "resourceGroup"=""
"ts"="2026-04-02 19:42:29.228630" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-65e261cf588b9f2637c69902ef65386ae7e651e35485a39612d668d6a3309cde.yaml.gz"
"ts"="2026-04-02 19:42:29.228771" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-65e261cf588b9f2637c69902ef65386ae7e651e35485a39612d668d6a3309cde.yaml.gz"
"ts"="2026-04-02 19:42:29.228877" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 19:42:29.231553" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-18"
"ts"="2026-04-02 19:42:29.231567" "level"=0 "msg"="collecting deployments" "resourceGroup"="private-keyvault-w6qm7t"
"ts"="2026-04-02 19:42:29.323497" "level"=0 "msg"="collecting operations" "resourceGroup"="private-keyvault-w6qm7t" "deployment"="pooledNonMsiScopedAssignments-had2isw6vkgre"
"ts"="2026-04-02 19:42:29.375537" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-18" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 19:42:29.376346" "level"=0 "msg"="collecting operations" "resourceGroup"="private-keyvault-w6qm7t" "deployment"="customer-infra-private-kv-cluster-brmrkf"
"ts"="2026-04-02 19:42:29.521817" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-18" "deployment"="msi-deployment-18"
"ts"="2026-04-02 19:42:29.680009" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 19:42:29.680123" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 19:42:29.680252" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="private-keyvault-w6qm7t"
Command Error: signal: killed
Deserializaion Error: no output from commandERROR CODE: DeploymentNotFound Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
<*fmt.wrapError | 0xc0010281e0>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-subnet-reuse-fnhchx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-subnet-reuse-fnhchx/operationStatuses/08584264532945813091
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: DeploymentNotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "DeploymentNotFound",
"message": "Deployment 'mi-cluster-subnet-reuse-fnhchx' could not be found."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-subnet-reuse-fnhchx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-subnet-reuse-fnhchx/operationStatuses/08584264532945813091\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DeploymentNotFound\",\n \"message\": \"Deployment 'mi-cluster-subnet-reuse-fnhchx' could not be found.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0010281a0>{
msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-subnet-reuse-fnhchx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-subnet-reuse-fnhchx/operationStatuses/08584264532945813091\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DeploymentNotFound\",\n \"message\": \"Deployment 'mi-cluster-subnet-reuse-fnhchx' could not be found.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc001028180>{
msg: "failed waiting for deployment \"mi-cluster-subnet-reuse-fnhchx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-subnet-reuse-fnhchx/operationStatuses/08584264532945813091\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DeploymentNotFound\",\n \"message\": \"Deployment 'mi-cluster-subnet-reuse-fnhchx' could not be found.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0006421b0>{
ErrorCode: "DeploymentNotFound",
StatusCode: 404,
RawResponse: {
Status: "404 Not Found",
StatusCode: 404,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"],
"Date": [
"Thu, 02 Apr 2026 18:13:11 GMT",
],
"X-Ms-Failure-Cause": ["gateway"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"],
"X-Ms-Request-Id": [
"4af1ec23-5e77-436c-a930-c88c039bbaf2",
],
"X-Ms-Correlation-Request-Id": [
"4af1ec23-5e77-436c-a930-c88c039bbaf2",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Content-Length": ["115"],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
<*fmt.wrapError | 0xc0010281e0>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-subnet-reuse-fnhchx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-subnet-reuse-fnhchx/operationStatuses/08584264532945813091
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: DeploymentNotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "DeploymentNotFound",
"message": "Deployment 'mi-cluster-subnet-reuse-fnhchx' could not be found."
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-subnet-reuse-fnhchx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-subnet-reuse-fnhchx/operationStatuses/08584264532945813091\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DeploymentNotFound\",\n \"message\": \"Deployment 'mi-cluster-subnet-reuse-fnhchx' could not be found.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0010281a0>{
msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-subnet-reuse-fnhchx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-subnet-reuse-fnhchx/operationStatuses/08584264532945813091\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DeploymentNotFound\",\n \"message\": \"Deployment 'mi-cluster-subnet-reuse-fnhchx' could not be found.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc001028180>{
msg: "failed waiting for deployment \"mi-cluster-subnet-reuse-fnhchx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-subnet-reuse-fnhchx/operationStatuses/08584264532945813091\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"DeploymentNotFound\",\n \"message\": \"Deployment 'mi-cluster-subnet-reuse-fnhchx' could not be found.\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0006421b0>{
ErrorCode: "DeploymentNotFound",
StatusCode: 404,
RawResponse: {
Status: "404 Not Found",
StatusCode: 404,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"],
"Date": [
"Thu, 02 Apr 2026 18:13:11 GMT",
],
"X-Ms-Failure-Cause": ["gateway"],
"X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"],
"X-Ms-Request-Id": [
"4af1ec23-5e77-436c-a930-c88c039bbaf2",
],
"X-Ms-Correlation-Request-Id": [
"4af1ec23-5e77-436c-a930-c88c039bbaf2",
],
"Strict-Transport-Security": [
"max-age=31536000; includeSubDomains",
],
"Cache-Control": ["no-cache"],
"Pragma": ["no-cache"],
"Content-Length": ["115"],
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredInterrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User |
|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
e2e | failed | n/a | 2 | e2e (2)Interrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User Interrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User |
|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
n/a | passed | n/a | 0 | n/a | |
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
e2e | failed | n/a | 2 | e2e (2)Interrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User Interrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User |
|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
e2e | failed | n/a | 2 | e2e (2)Interrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User Interrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User |
|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
n/a | passed | n/a | 0 | n/a | |
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
e2e | failed | n/a | 4 | e2e (4)Interrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User Interrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User Interrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User Interrupted by User Show raw failurefail [:0]: Interrupted by User fail [:0]: Interrupted by User |
|
| periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR |
e2e, unknown | failed | n/a | 30 | e2e (29)Deserializaion Error: no output from command Show raw failureRunning Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775093407 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:67
STEP: creating a resource group @ 04/02/26 01:30:11.709
"ts"="2026-04-02 01:30:11.709287" "level"=0 "msg"="creating resource group" "resourceGroup"="ephemeral-osdisk-74r57q"
STEP: creating cluster parameters @ 04/02/26 01:30:15.537
"ts"="2026-04-02 01:30:15.537817" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-04-02 01:30:15.537832" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources (infrastructure and managed identities) @ 04/02/26 01:30:15.537
STEP: creating the HCP cluster @ 04/02/26 01:31:11.592
"ts"="2026-04-02 01:31:11.592927" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="ephemeral-disk" "resourceGroup"="ephemeral-osdisk-74r57q"
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112 @ 04/02/26 02:16:11.595
"ts"="2026-04-02 02:16:11.596087" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 02:16:11.596281" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-ephemeral-disk-6gkhgv" "resourceGroup"="ephemeral-osdisk-74r57q"
"ts"="2026-04-02 02:16:11.767154" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-ephemeral-disk-6gkhgv" "resourceGroup"=""
"ts"="2026-04-02 02:16:13.253101" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-a54a7b95d75fb7a5c86e976d7b6a232359fe80ddc62a98537cfa2cfbd9f24bac.yaml.gz"
"ts"="2026-04-02 02:16:13.253248" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-a54a7b95d75fb7a5c86e976d7b6a232359fe80ddc62a98537cfa2cfbd9f24bac.yaml.gz"
"ts"="2026-04-02 02:16:13.253328" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 02:16:13.255519" "level"=0 "msg"="collecting deployments" "resourceGroup"="ephemeral-osdisk-74r57q"
"ts"="2026-04-02 02:16:13.255507" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-128"
"ts"="2026-04-02 02:16:13.391863" "level"=0 "msg"="collecting operations" "resourceGroup"="ephemeral-osdisk-74r57q" "deployment"="pooledNonMsiScopedAssignments-6xts4ycasve7q"
"ts"="2026-04-02 02:16:13.480661" "level"=0 "msg"="collecting operations" "resourceGroup"="ephemeral-osdisk-74r57q" "deployment"="customer-infra-ephemeral-disk-6gkhgv"
"ts"="2026-04-02 02:16:13.690527" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-128" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 02:16:13.855407" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-128" "deployment"="msi-deployment-128"
"ts"="2026-04-02 02:16:13.997026" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 02:16:13.997138" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 02:16:13.997296" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="ephemeral-osdisk-74r57q"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775093407 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:67
STEP: creating a resource group @ 04/02/26 01:30:11.709
"ts"="2026-04-02 01:30:11.709287" "level"=0 "msg"="creating resource group" "resourceGroup"="ephemeral-osdisk-74r57q"
STEP: creating cluster parameters @ 04/02/26 01:30:15.537
"ts"="2026-04-02 01:30:15.537817" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-04-02 01:30:15.537832" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources (infrastructure and managed identities) @ 04/02/26 01:30:15.537
STEP: creating the HCP cluster @ 04/02/26 01:31:11.592
"ts"="2026-04-02 01:31:11.592927" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="ephemeral-disk" "resourceGroup"="ephemeral-osdisk-74r57q"
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112 @ 04/02/26 02:16:11.595
"ts"="2026-04-02 02:16:11.596087" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 02:16:11.596281" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-ephemeral-disk-6gkhgv" "resourceGroup"="ephemeral-osdisk-74r57q"
"ts"="2026-04-02 02:16:11.767154" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-ephemeral-disk-6gkhgv" "resourceGroup"=""
"ts"="2026-04-02 02:16:13.253101" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-a54a7b95d75fb7a5c86e976d7b6a232359fe80ddc62a98537cfa2cfbd9f24bac.yaml.gz"
"ts"="2026-04-02 02:16:13.253248" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-a54a7b95d75fb7a5c86e976d7b6a232359fe80ddc62a98537cfa2cfbd9f24bac.yaml.gz"
"ts"="2026-04-02 02:16:13.253328" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 02:16:13.255519" "level"=0 "msg"="collecting deployments" "resourceGroup"="ephemeral-osdisk-74r57q"
"ts"="2026-04-02 02:16:13.255507" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-128"
"ts"="2026-04-02 02:16:13.391863" "level"=0 "msg"="collecting operations" "resourceGroup"="ephemeral-osdisk-74r57q" "deployment"="pooledNonMsiScopedAssignments-6xts4ycasve7q"
"ts"="2026-04-02 02:16:13.480661" "level"=0 "msg"="collecting operations" "resourceGroup"="ephemeral-osdisk-74r57q" "deployment"="customer-infra-ephemeral-disk-6gkhgv"
"ts"="2026-04-02 02:16:13.690527" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-128" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 02:16:13.855407" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-128" "deployment"="msi-deployment-128"
"ts"="2026-04-02 02:16:13.997026" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 02:16:13.997138" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 02:16:13.997296" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="ephemeral-osdisk-74r57q"
Command Error: signal: killed
Deserializaion Error: no output from commandERROR CODE: InternalServerError; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: Unexpected error:
<*fmt.wrapError | 0xc000bea800>:
failed to create HCP cluster np-version-upgrade-cluster-z8jp8j: failed waiting for cluster="np-version-upgrade-cluster-z8jp8j" in resourcegroup="rg-np-version-upgrade-z8jp8j-brqz27" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0",
"name": "3ad28075-f0f4-45dd-b143-f32346eb45d0",
"status": "Failed",
"startTime": "2026-04-02T01:33:37.647438148Z",
"endTime": "2026-04-02T01:34:08.890577638Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg-np-upgrade-z8jp8j' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b531933373' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster np-version-upgrade-cluster-z8jp8j: failed waiting for cluster=\"np-version-upgrade-cluster-z8jp8j\" in resourcegroup=\"rg-np-version-upgrade-z8jp8j-brqz27\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0\",\n \"name\": \"3ad28075-f0f4-45dd-b143-f32346eb45d0\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T01:33:37.647438148Z\",\n \"endTime\": \"2026-04-02T01:34:08.890577638Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-np-upgrade-z8jp8j' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b531933373' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000bea7c0>{
msg: "failed waiting for cluster=\"np-version-upgrade-cluster-z8jp8j\" in resourcegroup=\"rg-np-version-upgrade-z8jp8j-brqz27\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0\",\n \"name\": \"3ad28075-f0f4-45dd-b143-f32346eb45d0\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T01:33:37.647438148Z\",\n \"endTime\": \"2026-04-02T01:34:08.890577638Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-np-upgrade-z8jp8j' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: Unexpected error:
<*fmt.wrapError | 0xc000bea800>:
failed to create HCP cluster np-version-upgrade-cluster-z8jp8j: failed waiting for cluster="np-version-upgrade-cluster-z8jp8j" in resourcegroup="rg-np-version-upgrade-z8jp8j-brqz27" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0",
"name": "3ad28075-f0f4-45dd-b143-f32346eb45d0",
"status": "Failed",
"startTime": "2026-04-02T01:33:37.647438148Z",
"endTime": "2026-04-02T01:34:08.890577638Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg-np-upgrade-z8jp8j' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b531933373' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster np-version-upgrade-cluster-z8jp8j: failed waiting for cluster=\"np-version-upgrade-cluster-z8jp8j\" in resourcegroup=\"rg-np-version-upgrade-z8jp8j-brqz27\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0\",\n \"name\": \"3ad28075-f0f4-45dd-b143-f32346eb45d0\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T01:33:37.647438148Z\",\n \"endTime\": \"2026-04-02T01:34:08.890577638Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-np-upgrade-z8jp8j' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b531933373' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000bea7c0>{
msg: "failed waiting for cluster=\"np-version-upgrade-cluster-z8jp8j\" in resourcegroup=\"rg-np-version-upgrade-z8jp8j-brqz27\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0\",\n \"name\": \"3ad28075-f0f4-45dd-b143-f32346eb45d0\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T01:33:37.647438148Z\",\n \"endTime\": \"2026-04-02T01:34:08.890577638Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-np-upgrade-z8jp8j' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredtimeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:141]: Unexpected error:
<*fmt.wrapErrors | 0xc0011c08a0>:
failed waiting for cluster="cilium-cluster" in resourcegroup="e2e-cilium-dhfb8w" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group e2e-cilium-dhfb8w, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cilium-cluster\" in resourcegroup=\"e2e-cilium-dhfb8w\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group e2e-cilium-dhfb8w, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0011434e0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group e2e-cilium-dhfb8w",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:141]: Unexpected error:
<*fmt.wrapErrors | 0xc0011c08a0>:
failed waiting for cluster="cilium-cluster" in resourcegroup="e2e-cilium-dhfb8w" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group e2e-cilium-dhfb8w, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cilium-cluster\" in resourcegroup=\"e2e-cilium-dhfb8w\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group e2e-cilium-dhfb8w, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0011434e0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group e2e-cilium-dhfb8w",
},
<context.deadlineExceededError>{},
],
}
occurredDeserializaion Error: no output from command Show raw failureRunning Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775093608 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should not be able to perform various invalid operations on cluster resources [PreLaunchSetup:None, Positivity:Negative, Importance:Medium, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:41
STEP: creating a resource group @ 04/02/26 01:33:28.529
"ts"="2026-04-02 01:33:28.529807" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-negative-tests-q25j97"
STEP: creating cluster parameters @ 04/02/26 01:33:32.008
"ts"="2026-04-02 01:33:32.008858" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-04-02 01:33:32.008877" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 04/02/26 01:33:32.008
STEP: creating the HCP cluster @ 04/02/26 01:34:28.975
"ts"="2026-04-02 01:34:28.975113" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="negative-tests-cluster" "resourceGroup"="rg-negative-tests-q25j97"
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:117 @ 04/02/26 02:19:28.976
"ts"="2026-04-02 02:19:28.976618" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 02:19:28.976714" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-negative-tests-cluster-w42dxb" "resourceGroup"="rg-negative-tests-q25j97"
"ts"="2026-04-02 02:19:29.118328" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-negative-tests-cluster-w42dxb" "resourceGroup"=""
"ts"="2026-04-02 02:19:30.642103" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz"
"ts"="2026-04-02 02:19:30.642232" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz"
"ts"="2026-04-02 02:19:30.642302" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 02:19:30.643746" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-91"
"ts"="2026-04-02 02:19:30.643775" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-negative-tests-q25j97"
"ts"="2026-04-02 02:19:30.801321" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-q25j97" "deployment"="pooledNonMsiScopedAssignments-oeayaalhj7wjy"
"ts"="2026-04-02 02:19:30.816599" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-91" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 02:19:30.874154" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-q25j97" "deployment"="customer-infra-negative-tests-cluster-w42dxb"
"ts"="2026-04-02 02:19:30.957443" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-91" "deployment"="msi-deployment-91"
"ts"="2026-04-02 02:19:31.103536" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 02:19:31.103654" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 02:19:31.103784" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-negative-tests-q25j97"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775093608 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should not be able to perform various invalid operations on cluster resources [PreLaunchSetup:None, Positivity:Negative, Importance:Medium, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:41
STEP: creating a resource group @ 04/02/26 01:33:28.529
"ts"="2026-04-02 01:33:28.529807" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-negative-tests-q25j97"
STEP: creating cluster parameters @ 04/02/26 01:33:32.008
"ts"="2026-04-02 01:33:32.008858" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-04-02 01:33:32.008877" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 04/02/26 01:33:32.008
STEP: creating the HCP cluster @ 04/02/26 01:34:28.975
"ts"="2026-04-02 01:34:28.975113" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="negative-tests-cluster" "resourceGroup"="rg-negative-tests-q25j97"
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:117 @ 04/02/26 02:19:28.976
"ts"="2026-04-02 02:19:28.976618" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 02:19:28.976714" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-negative-tests-cluster-w42dxb" "resourceGroup"="rg-negative-tests-q25j97"
"ts"="2026-04-02 02:19:29.118328" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-negative-tests-cluster-w42dxb" "resourceGroup"=""
"ts"="2026-04-02 02:19:30.642103" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz"
"ts"="2026-04-02 02:19:30.642232" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz"
"ts"="2026-04-02 02:19:30.642302" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 02:19:30.643746" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-91"
"ts"="2026-04-02 02:19:30.643775" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-negative-tests-q25j97"
"ts"="2026-04-02 02:19:30.801321" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-q25j97" "deployment"="pooledNonMsiScopedAssignments-oeayaalhj7wjy"
"ts"="2026-04-02 02:19:30.816599" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-91" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 02:19:30.874154" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-q25j97" "deployment"="customer-infra-negative-tests-cluster-w42dxb"
"ts"="2026-04-02 02:19:30.957443" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-91" "deployment"="msi-deployment-91"
"ts"="2026-04-02 02:19:31.103536" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 02:19:31.103654" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 02:19:31.103784" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-negative-tests-q25j97"
Command Error: signal: killed
Deserializaion Error: no output from commandtimeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
<*fmt.wrapErrors | 0xc000ff1020>:
failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-txvwqn, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-txvwqn" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-txvwqn, error: context deadline exceeded
{
msg: "failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-txvwqn, error: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-txvwqn\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-txvwqn, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ccfdd0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-txvwqn",
},
<*fmt.wrapErrors | 0xc000ff0ff0>{
msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-txvwqn\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-txvwqn, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ccfdd0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-txvwqn",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
<*fmt.wrapErrors | 0xc000ff1020>:
failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-txvwqn, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-txvwqn" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-txvwqn, error: context deadline exceeded
{
msg: "failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-txvwqn, error: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-txvwqn\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-txvwqn, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ccfdd0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-txvwqn",
},
<*fmt.wrapErrors | 0xc000ff0ff0>{
msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-txvwqn\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-txvwqn, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ccfdd0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-txvwqn",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredERROR CODE: InternalServerError; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
<*fmt.wrapError | 0xc000894f20>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-8sxmq6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587",
"name": "a300f2ba-df79-4404-a4a1-aae22a4d2587",
"status": "Failed",
"startTime": "2026-04-02T00:22:54.935266426Z",
"endTime": "2026-04-02T00:23:25.288802016Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-8sxmq6\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587\",\n \"name\": \"a300f2ba-df79-4404-a4a1-aae22a4d2587\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T00:22:54.935266426Z\",\n \"endTime\": \"2026-04-02T00:23:25.288802016Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000894f00>{
msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-8sxmq6\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587\",\n \"name\": \"a300f2ba-df79-4404-a4a1-aae22a4d2587\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T00:22:54.935266426Z\",\n \"endTime\": \"2026-04-02T00:23:25.288802016Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authori...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
<*fmt.wrapError | 0xc000894f20>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-8sxmq6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587",
"name": "a300f2ba-df79-4404-a4a1-aae22a4d2587",
"status": "Failed",
"startTime": "2026-04-02T00:22:54.935266426Z",
"endTime": "2026-04-02T00:23:25.288802016Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-8sxmq6\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587\",\n \"name\": \"a300f2ba-df79-4404-a4a1-aae22a4d2587\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T00:22:54.935266426Z\",\n \"endTime\": \"2026-04-02T00:23:25.288802016Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000894f00>{
msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-8sxmq6\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587\",\n \"name\": \"a300f2ba-df79-4404-a4a1-aae22a4d2587\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T00:22:54.935266426Z\",\n \"endTime\": \"2026-04-02T00:23:25.288802016Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authori...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredtimeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
<*fmt.wrapErrors | 0xc0017000c0>:
failed to create HCP cluster ea-listrvdprm, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listrvdprm in resource group ea-list-7cqq2d, error: failed waiting for cluster="ea-listrvdprm" in resourcegroup="ea-list-7cqq2d" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listrvdprm in resource group ea-list-7cqq2d, error: context deadline exceeded
{
msg: "failed to create HCP cluster ea-listrvdprm, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listrvdprm in resource group ea-list-7cqq2d, error: failed waiting for cluster=\"ea-listrvdprm\" in resourcegroup=\"ea-list-7cqq2d\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listrvdprm in resource group ea-list-7cqq2d, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000b04540>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listrvdprm in resource group ea-list-7cqq2d",
},
<*fmt.wrapErrors | 0xc001700090>{
msg: "failed waiting for cluster=\"ea-listrvdprm\" in resourcegroup=\"ea-list-7cqq2d\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listrvdprm in resource group ea-list-7cqq2d, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000b04540>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listrvdprm in resource group ea-list-7cqq2d",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
<*fmt.wrapErrors | 0xc0017000c0>:
failed to create HCP cluster ea-listrvdprm, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listrvdprm in resource group ea-list-7cqq2d, error: failed waiting for cluster="ea-listrvdprm" in resourcegroup="ea-list-7cqq2d" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listrvdprm in resource group ea-list-7cqq2d, error: context deadline exceeded
{
msg: "failed to create HCP cluster ea-listrvdprm, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listrvdprm in resource group ea-list-7cqq2d, error: failed waiting for cluster=\"ea-listrvdprm\" in resourcegroup=\"ea-list-7cqq2d\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listrvdprm in resource group ea-list-7cqq2d, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000b04540>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listrvdprm in resource group ea-list-7cqq2d",
},
<*fmt.wrapErrors | 0xc001700090>{
msg: "failed waiting for cluster=\"ea-listrvdprm\" in resourcegroup=\"ea-list-7cqq2d\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listrvdprm in resource group ea-list-7cqq2d, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000b04540>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listrvdprm in resource group ea-list-7cqq2d",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredtimeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
<*fmt.wrapErrors | 0xc0011e0d50>:
failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-9sfb27, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9sfb27" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-9sfb27, error: context deadline exceeded
{
msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-9sfb27, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-9sfb27\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-9sfb27, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00119cac0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-9sfb27",
},
<*fmt.wrapErrors | 0xc0011e0d20>{
msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-9sfb27\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-9sfb27, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00119cac0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-9sfb27",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
<*fmt.wrapErrors | 0xc0011e0d50>:
failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-9sfb27, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9sfb27" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-9sfb27, error: context deadline exceeded
{
msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-9sfb27, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-9sfb27\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-9sfb27, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00119cac0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-9sfb27",
},
<*fmt.wrapErrors | 0xc0011e0d20>{
msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-9sfb27\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-9sfb27, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00119cac0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-9sfb27",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredERROR CODE: InternalServerError; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
<*fmt.wrapError | 0xc00078d420>:
failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-p4zvhp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29",
"name": "f32ace59-a2fb-42f6-9421-b40f5132fd29",
"status": "Failed",
"startTime": "2026-04-02T00:27:03.073725945Z",
"endTime": "2026-04-02T00:27:41.510479828Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-p4zvhp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29\",\n \"name\": \"f32ace59-a2fb-42f6-9421-b40f5132fd29\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T00:27:03.073725945Z\",\n \"endTime\": \"2026-04-02T00:27:41.510479828Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00078d3c0>{
msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-p4zvhp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29\",\n \"name\": \"f32ace59-a2fb-42f6-9421-b40f5132fd29\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T00:27:03.073725945Z\",\n \"endTime\": \"2026-04-02T00:27:41.510479828Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not ha...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
<*fmt.wrapError | 0xc00078d420>:
failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-p4zvhp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29",
"name": "f32ace59-a2fb-42f6-9421-b40f5132fd29",
"status": "Failed",
"startTime": "2026-04-02T00:27:03.073725945Z",
"endTime": "2026-04-02T00:27:41.510479828Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-p4zvhp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29\",\n \"name\": \"f32ace59-a2fb-42f6-9421-b40f5132fd29\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T00:27:03.073725945Z\",\n \"endTime\": \"2026-04-02T00:27:41.510479828Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00078d3c0>{
msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-p4zvhp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29\",\n \"name\": \"f32ace59-a2fb-42f6-9421-b40f5132fd29\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T00:27:03.073725945Z\",\n \"endTime\": \"2026-04-02T00:27:41.510479828Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not ha...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredDeserializaion Error: no output from command Show raw failure> github.com/Azure/ARO-HCP/test/e2e.init.func17.2({0x7f395865ae60, 0xc000a84360})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126
|
| By("waiting for first cluster to complete creation")
> _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{
| Frequency: framework.StandardPollInterval,
| })
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8cc2c80?, 0xc000a84360?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 73
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 04/02/26 01:51:54.509
"ts"="2026-04-02 01:51:54.509749" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 01:51:54.509817" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-68kpl4" "resourceGroup"="customer-rg-wz4brf"
"ts"="2026-04-02 01:51:54.669907" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-68kpl4" "resourceGroup"=""
"ts"="2026-04-02 01:51:55.462809" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-nw72jh" "resourceGroup"="customer-rg-wz4brf"
"ts"="2026-04-02 01:51:55.588392" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-nw72jh" "resourceGroup"=""
"ts"="2026-04-02 01:51:57.012940" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
"ts"="2026-04-02 01:51:57.013048" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
"ts"="2026-04-02 01:51:57.013105" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 01:51:57.014271" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-109"
"ts"="2026-04-02 01:51:57.014292" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-123"
"ts"="2026-04-02 01:51:57.014281" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-113"
"ts"="2026-04-02 01:51:57.014340" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-wz4brf"
"ts"="2026-04-02 01:51:57.247683" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-wz4brf" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32"
"ts"="2026-04-02 01:51:57.326543" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-wz4brf" "deployment"="customer-infra-basic-hcp-cluster2-nw72jh"
"ts"="2026-04-02 01:51:57.361557" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-113" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 01:51:57.433058" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-wz4brf" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
"ts"="2026-04-02 01:51:57.501750" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-113" "deployment"="msi-deployment-113"
"ts"="2026-04-02 01:51:57.509668" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-wz4brf" "deployment"="customer-infra-basic-hcp-cluster-68kpl4"
"ts"="2026-04-02 01:51:57.534043" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-123" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 01:51:57.557500" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-109" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 01:51:57.665194" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-123" "deployment"="msi-deployment-123"
"ts"="2026-04-02 01:51:57.740187" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-109" "deployment"="msi-deployment-109"
"ts"="2026-04-02 01:51:57.874853" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 01:51:57.874990" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 01:51:57.875105" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-wz4brf"
Command Error: signal: killed
Deserializaion Error: no output from command
> github.com/Azure/ARO-HCP/test/e2e.init.func17.2({0x7f395865ae60, 0xc000a84360})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126
|
| By("waiting for first cluster to complete creation")
> _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{
| Frequency: framework.StandardPollInterval,
| })
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8cc2c80?, 0xc000a84360?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 73
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 04/02/26 01:51:54.509
"ts"="2026-04-02 01:51:54.509749" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 01:51:54.509817" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-68kpl4" "resourceGroup"="customer-rg-wz4brf"
"ts"="2026-04-02 01:51:54.669907" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-68kpl4" "resourceGroup"=""
"ts"="2026-04-02 01:51:55.462809" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-nw72jh" "resourceGroup"="customer-rg-wz4brf"
"ts"="2026-04-02 01:51:55.588392" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-nw72jh" "resourceGroup"=""
"ts"="2026-04-02 01:51:57.012940" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
"ts"="2026-04-02 01:51:57.013048" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
"ts"="2026-04-02 01:51:57.013105" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 01:51:57.014271" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-109"
"ts"="2026-04-02 01:51:57.014292" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-123"
"ts"="2026-04-02 01:51:57.014281" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-113"
"ts"="2026-04-02 01:51:57.014340" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-wz4brf"
"ts"="2026-04-02 01:51:57.247683" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-wz4brf" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32"
"ts"="2026-04-02 01:51:57.326543" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-wz4brf" "deployment"="customer-infra-basic-hcp-cluster2-nw72jh"
"ts"="2026-04-02 01:51:57.361557" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-113" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 01:51:57.433058" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-wz4brf" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
"ts"="2026-04-02 01:51:57.501750" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-113" "deployment"="msi-deployment-113"
"ts"="2026-04-02 01:51:57.509668" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-wz4brf" "deployment"="customer-infra-basic-hcp-cluster-68kpl4"
"ts"="2026-04-02 01:51:57.534043" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-123" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 01:51:57.557500" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-109" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 01:51:57.665194" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-123" "deployment"="msi-deployment-123"
"ts"="2026-04-02 01:51:57.740187" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-109" "deployment"="msi-deployment-109"
"ts"="2026-04-02 01:51:57.874853" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 01:51:57.874990" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 01:51:57.875105" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-wz4brf"
Command Error: signal: killed
Deserializaion Error: no output from commandERROR CODE: InternalServerError; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
<*fmt.wrapError | 0xc0010502c0>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-8xlw7k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3",
"name": "3484e971-8275-4ae3-a76f-65d592316cf3",
"status": "Failed",
"startTime": "2026-04-02T01:29:14.718319727Z",
"endTime": "2026-04-02T01:30:18.264779861Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b531933373' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-8xlw7k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3\",\n \"name\": \"3484e971-8275-4ae3-a76f-65d592316cf3\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T01:29:14.718319727Z\",\n \"endTime\": \"2026-04-02T01:30:18.264779861Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b531933373' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0010502a0>{
msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-8xlw7k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3\",\n \"name\": \"3484e971-8275-4ae3-a76f-65d592316cf3\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T01:29:14.718319727Z\",\n \"endTime\": \"2026-04-02T01:30:18.264779861Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b5319333...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
<*fmt.wrapError | 0xc0010502c0>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-8xlw7k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3",
"name": "3484e971-8275-4ae3-a76f-65d592316cf3",
"status": "Failed",
"startTime": "2026-04-02T01:29:14.718319727Z",
"endTime": "2026-04-02T01:30:18.264779861Z",
"error": {
"code": "InternalServerError",
"message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b531933373' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-8xlw7k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3\",\n \"name\": \"3484e971-8275-4ae3-a76f-65d592316cf3\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T01:29:14.718319727Z\",\n \"endTime\": \"2026-04-02T01:30:18.264779861Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b531933373' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0010502a0>{
msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-8xlw7k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3\",\n \"name\": \"3484e971-8275-4ae3-a76f-65d592316cf3\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T01:29:14.718319727Z\",\n \"endTime\": \"2026-04-02T01:30:18.264779861Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b5319333...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredDeserializaion Error: no output from command Show raw failureinkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:199
> github.com/onsi/ginkgo/v2/internal.(*group).run(0xc000db0b98, {0xc000c64f60?, 0x0?, 0x0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349
> github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000932a88, {0x0, 0x0}, {0xc1de280, 0x0, 0x0}, {0xc0000bc064, 0x2a}, 0x0, {0xc000c64f40, ...})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496
> github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc0000bc064, 0x2a}, ...)
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68
github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2d45a85?, 0xc000c00c60?})
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/ginkgo/util.go:85
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x8caa8c8, 0xc000b5cff0}, 0xc000b44900, 0x1)
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:596
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:237
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:230
goroutine 99 [select]
> github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0xc000b44900?)
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:131
> github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 98
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:128
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137 @ 04/02/26 01:51:54.509
"ts"="2026-04-02 01:51:54.510196" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 01:51:54.510248" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-zstream-4-19-zxltj9-ncfxtn" "resourceGroup"="rg-zstream-upgrade-4-19-d4r4n7"
"ts"="2026-04-02 01:51:55.897221" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-zstream-4-19-zxltj9-ncfxtn" "resourceGroup"=""
"ts"="2026-04-02 01:51:58.433435" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-0c1d721d40414cddab3a52b9d7add0203efdd55a4c816fc13c12379212f5cea0.yaml.gz"
"ts"="2026-04-02 01:51:58.433576" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-0c1d721d40414cddab3a52b9d7add0203efdd55a4c816fc13c12379212f5cea0.yaml.gz"
"ts"="2026-04-02 01:51:58.433670" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 01:51:58.435153" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-100"
"ts"="2026-04-02 01:51:58.435190" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-zstream-upgrade-4-19-d4r4n7"
"ts"="2026-04-02 01:51:58.537030" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-zstream-upgrade-4-19-d4r4n7" "deployment"="pooledNonMsiScopedAssignments-fldd5huuy6tgg"
"ts"="2026-04-02 01:51:58.617554" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-100" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 01:51:58.695091" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-zstream-upgrade-4-19-d4r4n7" "deployment"="customer-infra-cluster-zstream-4-19-zxltj9-ncfxtn"
"ts"="2026-04-02 01:51:58.752078" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-100" "deployment"="msi-deployment-100"
"ts"="2026-04-02 01:51:58.906467" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 01:51:58.906589" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 01:51:58.906768" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-zstream-upgrade-4-19-d4r4n7"
Command Error: signal: killed
Deserializaion Error: no output from command
inkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:199
> github.com/onsi/ginkgo/v2/internal.(*group).run(0xc000db0b98, {0xc000c64f60?, 0x0?, 0x0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349
> github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000932a88, {0x0, 0x0}, {0xc1de280, 0x0, 0x0}, {0xc0000bc064, 0x2a}, 0x0, {0xc000c64f40, ...})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496
> github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc0000bc064, 0x2a}, ...)
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68
github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2d45a85?, 0xc000c00c60?})
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/ginkgo/util.go:85
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x8caa8c8, 0xc000b5cff0}, 0xc000b44900, 0x1)
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:596
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:237
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:230
goroutine 99 [select]
> github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0xc000b44900?)
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:131
> github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 98
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:128
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137 @ 04/02/26 01:51:54.509
"ts"="2026-04-02 01:51:54.510196" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 01:51:54.510248" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-zstream-4-19-zxltj9-ncfxtn" "resourceGroup"="rg-zstream-upgrade-4-19-d4r4n7"
"ts"="2026-04-02 01:51:55.897221" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-zstream-4-19-zxltj9-ncfxtn" "resourceGroup"=""
"ts"="2026-04-02 01:51:58.433435" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-0c1d721d40414cddab3a52b9d7add0203efdd55a4c816fc13c12379212f5cea0.yaml.gz"
"ts"="2026-04-02 01:51:58.433576" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-0c1d721d40414cddab3a52b9d7add0203efdd55a4c816fc13c12379212f5cea0.yaml.gz"
"ts"="2026-04-02 01:51:58.433670" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 01:51:58.435153" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-100"
"ts"="2026-04-02 01:51:58.435190" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-zstream-upgrade-4-19-d4r4n7"
"ts"="2026-04-02 01:51:58.537030" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-zstream-upgrade-4-19-d4r4n7" "deployment"="pooledNonMsiScopedAssignments-fldd5huuy6tgg"
"ts"="2026-04-02 01:51:58.617554" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-100" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 01:51:58.695091" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-zstream-upgrade-4-19-d4r4n7" "deployment"="customer-infra-cluster-zstream-4-19-zxltj9-ncfxtn"
"ts"="2026-04-02 01:51:58.752078" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-100" "deployment"="msi-deployment-100"
"ts"="2026-04-02 01:51:58.906467" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 01:51:58.906589" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 01:51:58.906768" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-zstream-upgrade-4-19-d4r4n7"
Command Error: signal: killed
Deserializaion Error: no output from commandtimeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
<*fmt.wrapErrors | 0xc001533530>:
failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-pw2btq, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-pw2btq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-pw2btq, error: context deadline exceeded
{
msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-pw2btq, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-pw2btq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-pw2btq, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000f77d30>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-pw2btq",
},
<*fmt.wrapErrors | 0xc001533500>{
msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-pw2btq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-pw2btq, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000f77d30>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-pw2btq",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
<*fmt.wrapErrors | 0xc001533530>:
failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-pw2btq, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-pw2btq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-pw2btq, error: context deadline exceeded
{
msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-pw2btq, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-pw2btq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-pw2btq, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000f77d30>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-pw2btq",
},
<*fmt.wrapErrors | 0xc001533500>{
msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-pw2btq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-pw2btq, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000f77d30>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-pw2btq",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredtimeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
<*fmt.wrapErrors | 0xc001372f90>:
failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-wvwqlp, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-wvwqlp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-wvwqlp, error: context deadline exceeded
{
msg: "failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-wvwqlp, error: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-wvwqlp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-wvwqlp, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0013bd230>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-wvwqlp",
},
<*fmt.wrapErrors | 0xc001372f60>{
msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-wvwqlp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-wvwqlp, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0013bd230>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-wvwqlp",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
<*fmt.wrapErrors | 0xc001372f90>:
failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-wvwqlp, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-wvwqlp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-wvwqlp, error: context deadline exceeded
{
msg: "failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-wvwqlp, error: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-wvwqlp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-wvwqlp, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0013bd230>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-wvwqlp",
},
<*fmt.wrapErrors | 0xc001372f60>{
msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-wvwqlp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-wvwqlp, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0013bd230>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-wvwqlp",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredtimeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: Unexpected error:
<*fmt.wrapErrors | 0xc001104e10>:
failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-c4nrww, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-c4nrww" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-c4nrww, error: context deadline exceeded
{
msg: "failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-c4nrww, error: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-c4nrww\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-c4nrww, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0015dd0e0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-c4nrww",
},
<*fmt.wrapErrors | 0xc001104de0>{
msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-c4nrww\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-c4nrww, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0015dd0e0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-c4nrww",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: Unexpected error:
<*fmt.wrapErrors | 0xc001104e10>:
failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-c4nrww, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-c4nrww" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-c4nrww, error: context deadline exceeded
{
msg: "failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-c4nrww, error: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-c4nrww\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-c4nrww, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0015dd0e0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-c4nrww",
},
<*fmt.wrapErrors | 0xc001104de0>{
msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-c4nrww\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-c4nrww, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0015dd0e0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-c4nrww",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredDeserializaion Error: no output from command Show raw failureinkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:199
> github.com/onsi/ginkgo/v2/internal.(*group).run(0xc000d5cb98, {0xc000b0a9a0?, 0x0?, 0x0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349
> github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000d3a008, {0x0, 0x0}, {0xc1de280, 0x0, 0x0}, {0xc0000bc064, 0x2a}, 0x0, {0xc000b0a980, ...})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496
> github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc0000bc064, 0x2a}, ...)
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68
github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2d45a85?, 0xc000a92c60?})
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/ginkgo/util.go:85
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x8caa8c8, 0xc000913cc0}, 0xc001012f00, 0x1)
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:596
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:237
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:230
goroutine 88 [select]
> github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0xc001012f00?)
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:131
> github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 87
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:128
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137 @ 04/02/26 01:51:54.631
"ts"="2026-04-02 01:51:54.631696" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 01:51:54.631774" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-zstream-4-20-dcg5c9-xksvgn" "resourceGroup"="rg-zstream-upgrade-4-20-zcjg2t"
"ts"="2026-04-02 01:51:55.940564" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-zstream-4-20-dcg5c9-xksvgn" "resourceGroup"=""
"ts"="2026-04-02 01:51:57.236284" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-c1cc868701677ba9bad3c4e2620f4c229f47803f50af63ee5732ea435a94776c.yaml.gz"
"ts"="2026-04-02 01:51:57.236400" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-c1cc868701677ba9bad3c4e2620f4c229f47803f50af63ee5732ea435a94776c.yaml.gz"
"ts"="2026-04-02 01:51:57.236476" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 01:51:57.237670" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-142"
"ts"="2026-04-02 01:51:57.237687" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-zstream-upgrade-4-20-zcjg2t"
"ts"="2026-04-02 01:51:57.399756" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-142" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 01:51:57.542142" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-142" "deployment"="msi-deployment-142"
"ts"="2026-04-02 01:51:57.799719" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-zstream-upgrade-4-20-zcjg2t" "deployment"="pooledNonMsiScopedAssignments-ufgsuor5fm6ve"
"ts"="2026-04-02 01:51:58.700077" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-zstream-upgrade-4-20-zcjg2t" "deployment"="customer-infra-cluster-zstream-4-20-dcg5c9-xksvgn"
"ts"="2026-04-02 01:51:59.922401" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 01:51:59.922536" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 01:51:59.922704" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-zstream-upgrade-4-20-zcjg2t"
Command Error: signal: killed
Deserializaion Error: no output from command
inkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:199
> github.com/onsi/ginkgo/v2/internal.(*group).run(0xc000d5cb98, {0xc000b0a9a0?, 0x0?, 0x0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349
> github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000d3a008, {0x0, 0x0}, {0xc1de280, 0x0, 0x0}, {0xc0000bc064, 0x2a}, 0x0, {0xc000b0a980, ...})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496
> github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc0000bc064, 0x2a}, ...)
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68
github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2d45a85?, 0xc000a92c60?})
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/ginkgo/util.go:85
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x8caa8c8, 0xc000913cc0}, 0xc001012f00, 0x1)
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:596
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:237
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:230
goroutine 88 [select]
> github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0xc001012f00?)
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:131
> github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 87
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:128
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137 @ 04/02/26 01:51:54.631
"ts"="2026-04-02 01:51:54.631696" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 01:51:54.631774" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-zstream-4-20-dcg5c9-xksvgn" "resourceGroup"="rg-zstream-upgrade-4-20-zcjg2t"
"ts"="2026-04-02 01:51:55.940564" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-zstream-4-20-dcg5c9-xksvgn" "resourceGroup"=""
"ts"="2026-04-02 01:51:57.236284" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-c1cc868701677ba9bad3c4e2620f4c229f47803f50af63ee5732ea435a94776c.yaml.gz"
"ts"="2026-04-02 01:51:57.236400" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-c1cc868701677ba9bad3c4e2620f4c229f47803f50af63ee5732ea435a94776c.yaml.gz"
"ts"="2026-04-02 01:51:57.236476" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 01:51:57.237670" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-142"
"ts"="2026-04-02 01:51:57.237687" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-zstream-upgrade-4-20-zcjg2t"
"ts"="2026-04-02 01:51:57.399756" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-142" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 01:51:57.542142" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-142" "deployment"="msi-deployment-142"
"ts"="2026-04-02 01:51:57.799719" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-zstream-upgrade-4-20-zcjg2t" "deployment"="pooledNonMsiScopedAssignments-ufgsuor5fm6ve"
"ts"="2026-04-02 01:51:58.700077" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-zstream-upgrade-4-20-zcjg2t" "deployment"="customer-infra-cluster-zstream-4-20-dcg5c9-xksvgn"
"ts"="2026-04-02 01:51:59.922401" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 01:51:59.922536" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 01:51:59.922704" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-zstream-upgrade-4-20-zcjg2t"
Command Error: signal: killed
Deserializaion Error: no output from commandERROR CODE: InternalServerError; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: Unexpected error:
<*fmt.wrapError | 0xc00131a180>:
failed to create HCP cluster cluster-zstream-4-21-xvw7fn: failed waiting for cluster="cluster-zstream-4-21-xvw7fn" in resourcegroup="rg-zstream-upgrade-4-21-qf98rq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23",
"name": "66f6aa6d-0e58-4cf7-a817-132dc9b1db23",
"status": "Failed",
"startTime": "2026-04-02T01:34:57.397134314Z",
"endTime": "2026-04-02T01:36:07.715905327Z",
"error": {
"code": "InternalServerError",
"message": "Failed to get virtual network 'customer-vnet-zstream-xvw7fn' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client '0c358081-82b1-4f65-97bc-be8647a5b112' with object id 'b3c21201-bf1b-4718-b697-526dbc98e7e9' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-21-xvw7fn: failed waiting for cluster=\"cluster-zstream-4-21-xvw7fn\" in resourcegroup=\"rg-zstream-upgrade-4-21-qf98rq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23\",\n \"name\": \"66f6aa6d-0e58-4cf7-a817-132dc9b1db23\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T01:34:57.397134314Z\",\n \"endTime\": \"2026-04-02T01:36:07.715905327Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-zstream-xvw7fn' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client '0c358081-82b1-4f65-97bc-be8647a5b112' with object id 'b3c21201-bf1b-4718-b697-526dbc98e7e9' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00131a160>{
msg: "failed waiting for cluster=\"cluster-zstream-4-21-xvw7fn\" in resourcegroup=\"rg-zstream-upgrade-4-21-qf98rq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23\",\n \"name\": \"66f6aa6d-0e58-4cf7-a817-132dc9b1db23\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T01:34:57.397134314Z\",\n \"endTime\": \"2026-04-02T01:36:07.715905327Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-zstream-xvw7fn' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: Unexpected error:
<*fmt.wrapError | 0xc00131a180>:
failed to create HCP cluster cluster-zstream-4-21-xvw7fn: failed waiting for cluster="cluster-zstream-4-21-xvw7fn" in resourcegroup="rg-zstream-upgrade-4-21-qf98rq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23",
"name": "66f6aa6d-0e58-4cf7-a817-132dc9b1db23",
"status": "Failed",
"startTime": "2026-04-02T01:34:57.397134314Z",
"endTime": "2026-04-02T01:36:07.715905327Z",
"error": {
"code": "InternalServerError",
"message": "Failed to get virtual network 'customer-vnet-zstream-xvw7fn' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client '0c358081-82b1-4f65-97bc-be8647a5b112' with object id 'b3c21201-bf1b-4718-b697-526dbc98e7e9' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n"
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create HCP cluster cluster-zstream-4-21-xvw7fn: failed waiting for cluster=\"cluster-zstream-4-21-xvw7fn\" in resourcegroup=\"rg-zstream-upgrade-4-21-qf98rq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23\",\n \"name\": \"66f6aa6d-0e58-4cf7-a817-132dc9b1db23\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T01:34:57.397134314Z\",\n \"endTime\": \"2026-04-02T01:36:07.715905327Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-zstream-xvw7fn' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client '0c358081-82b1-4f65-97bc-be8647a5b112' with object id 'b3c21201-bf1b-4718-b697-526dbc98e7e9' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc00131a160>{
msg: "failed waiting for cluster=\"cluster-zstream-4-21-xvw7fn\" in resourcegroup=\"rg-zstream-upgrade-4-21-qf98rq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23\",\n \"name\": \"66f6aa6d-0e58-4cf7-a817-132dc9b1db23\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-04-02T01:34:57.397134314Z\",\n \"endTime\": \"2026-04-02T01:36:07.715905327Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-zstream-xvw7fn' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredDeserializaion Error: no output from command Show raw failureRunning Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775093289 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36
STEP: creating a resource group @ 04/02/26 01:28:12.111
"ts"="2026-04-02 01:28:12.111559" "level"=0 "msg"="creating resource group" "resourceGroup"="np-autoscaling-wxdqdk"
STEP: creating cluster parameters @ 04/02/26 01:28:15.958
"ts"="2026-04-02 01:28:15.958468" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-04-02 01:28:15.958480" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 04/02/26 01:28:15.958
STEP: creating the HCP cluster @ 04/02/26 01:29:12.294
"ts"="2026-04-02 01:29:12.294519" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-autoscale-cluster" "resourceGroup"="np-autoscaling-wxdqdk"
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83 @ 04/02/26 02:14:12.297
"ts"="2026-04-02 02:14:12.297853" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 02:14:12.297926" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-autoscale-cluster-bpqwfl" "resourceGroup"="np-autoscaling-wxdqdk"
"ts"="2026-04-02 02:14:12.443280" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-autoscale-cluster-bpqwfl" "resourceGroup"=""
"ts"="2026-04-02 02:14:13.765625" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
"ts"="2026-04-02 02:14:13.765760" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
"ts"="2026-04-02 02:14:13.765865" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 02:14:13.767015" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-140"
"ts"="2026-04-02 02:14:13.767036" "level"=0 "msg"="collecting deployments" "resourceGroup"="np-autoscaling-wxdqdk"
"ts"="2026-04-02 02:14:13.873433" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-wxdqdk" "deployment"="pooledNonMsiScopedAssignments-ssrwqrnhyyabs"
"ts"="2026-04-02 02:14:13.928981" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-140" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 02:14:13.948021" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-wxdqdk" "deployment"="customer-infra-np-autoscale-cluster-bpqwfl"
"ts"="2026-04-02 02:14:14.080153" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-140" "deployment"="msi-deployment-140"
"ts"="2026-04-02 02:14:14.225391" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 02:14:14.225510" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 02:14:14.225635" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="np-autoscaling-wxdqdk"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775093289 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36
STEP: creating a resource group @ 04/02/26 01:28:12.111
"ts"="2026-04-02 01:28:12.111559" "level"=0 "msg"="creating resource group" "resourceGroup"="np-autoscaling-wxdqdk"
STEP: creating cluster parameters @ 04/02/26 01:28:15.958
"ts"="2026-04-02 01:28:15.958468" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-04-02 01:28:15.958480" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 04/02/26 01:28:15.958
STEP: creating the HCP cluster @ 04/02/26 01:29:12.294
"ts"="2026-04-02 01:29:12.294519" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-autoscale-cluster" "resourceGroup"="np-autoscaling-wxdqdk"
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83 @ 04/02/26 02:14:12.297
"ts"="2026-04-02 02:14:12.297853" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 02:14:12.297926" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-autoscale-cluster-bpqwfl" "resourceGroup"="np-autoscaling-wxdqdk"
"ts"="2026-04-02 02:14:12.443280" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-autoscale-cluster-bpqwfl" "resourceGroup"=""
"ts"="2026-04-02 02:14:13.765625" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
"ts"="2026-04-02 02:14:13.765760" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
"ts"="2026-04-02 02:14:13.765865" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 02:14:13.767015" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-140"
"ts"="2026-04-02 02:14:13.767036" "level"=0 "msg"="collecting deployments" "resourceGroup"="np-autoscaling-wxdqdk"
"ts"="2026-04-02 02:14:13.873433" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-wxdqdk" "deployment"="pooledNonMsiScopedAssignments-ssrwqrnhyyabs"
"ts"="2026-04-02 02:14:13.928981" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-140" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 02:14:13.948021" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-wxdqdk" "deployment"="customer-infra-np-autoscale-cluster-bpqwfl"
"ts"="2026-04-02 02:14:14.080153" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-140" "deployment"="msi-deployment-140"
"ts"="2026-04-02 02:14:14.225391" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 02:14:14.225510" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 02:14:14.225635" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="np-autoscaling-wxdqdk"
Command Error: signal: killed
Deserializaion Error: no output from commandtimeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:135]: Unexpected error:
<*fmt.wrapErrors | 0xc0010d6660>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-mxj67k" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-mxj67k, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-mxj67k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-mxj67k, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000fcc6d0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-mxj67k",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:135]: Unexpected error:
<*fmt.wrapErrors | 0xc0010d6660>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-mxj67k" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-mxj67k, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-mxj67k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-mxj67k, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000fcc6d0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-mxj67k",
},
<context.deadlineExceededError>{},
],
}
occurredCluster provisioning failed Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:153]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:153]: Cluster provisioning failed timeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
<*fmt.wrapErrors | 0xc0009c8c90>:
failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zhd77m, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-zhd77m" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zhd77m, error: context deadline exceeded
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zhd77m, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-zhd77m\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zhd77m, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000d3ab60>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zhd77m",
},
<*fmt.wrapErrors | 0xc0009c8c60>{
msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-zhd77m\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zhd77m, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000d3ab60>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zhd77m",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
<*fmt.wrapErrors | 0xc0009c8c90>:
failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zhd77m, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-zhd77m" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zhd77m, error: context deadline exceeded
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zhd77m, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-zhd77m\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zhd77m, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000d3ab60>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zhd77m",
},
<*fmt.wrapErrors | 0xc0009c8c60>{
msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-zhd77m\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zhd77m, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000d3ab60>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-zhd77m",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredDeserializaion Error: no output from command Show raw failureining specs. Interrupt again to skip cleanup.
Here's a current progress report:
Customer should be able to create an HCP cluster (Spec Runtime: 1h29m59.852s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:38
In [It] (Node Runtime: 1h29m59.852s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:38
At [By Step] creating the HCP cluster (Step Runtime: 15m5.938s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:88
Spec Goroutine
goroutine 81 [select]
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8caa8c8, 0xc001586820}, 0x81d9580?)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8ce5ea0, {0x8caa8c8, 0xc001586820}, 0x0)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x8caa938?, 0xc000704380?}, {{0x8cc2ba8?, 0xc000e26280?}, 0xc00093c648?}, 0xc0000d7768, {0xc00168e690, 0x14}, {0x81f1f5c, 0x11}, ...)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:694
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x81b78f8, 0x4}, ...}, ...)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:346
> github.com/Azure/ARO-HCP/test/e2e.init.func18.2({0x7f585c2cf198, 0xc000e74e10})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:89
|
| By("creating the HCP cluster")
> err = tc.CreateHCPClusterFromParam(
| ctx,
| GinkgoLogr,
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8cc2c80?, 0xc000e74e10?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 78
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96 @ 04/02/26 01:51:54.509
"ts"="2026-04-02 01:51:54.509787" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 01:51:54.509849" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-qwz4qp" "resourceGroup"="basic-cluster-448hmt"
"ts"="2026-04-02 01:51:54.642128" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-qwz4qp" "resourceGroup"=""
"ts"="2026-04-02 01:51:55.916040" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-8d7dc92573349c6d0052ecc8bc03042b1bfdfade27aae309024a834a40d01d5c.yaml.gz"
"ts"="2026-04-02 01:51:55.916147" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-8d7dc92573349c6d0052ecc8bc03042b1bfdfade27aae309024a834a40d01d5c.yaml.gz"
"ts"="2026-04-02 01:51:55.916233" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 01:51:55.917374" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-135"
"ts"="2026-04-02 01:51:55.917390" "level"=0 "msg"="collecting deployments" "resourceGroup"="basic-cluster-448hmt"
"ts"="2026-04-02 01:51:56.033502" "level"=0 "msg"="collecting operations" "resourceGroup"="basic-cluster-448hmt" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
"ts"="2026-04-02 01:51:56.066101" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-135" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 01:51:56.118598" "level"=0 "msg"="collecting operations" "resourceGroup"="basic-cluster-448hmt" "deployment"="customer-infra-basic-hcp-cluster-qwz4qp"
"ts"="2026-04-02 01:51:56.406662" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-135" "deployment"="msi-deployment-135"
"ts"="2026-04-02 01:51:56.541695" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 01:51:56.541805" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 01:51:56.541940" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="basic-cluster-448hmt"
Command Error: signal: killed
Deserializaion Error: no output from command
ining specs. Interrupt again to skip cleanup.
Here's a current progress report:
Customer should be able to create an HCP cluster (Spec Runtime: 1h29m59.852s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:38
In [It] (Node Runtime: 1h29m59.852s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:38
At [By Step] creating the HCP cluster (Step Runtime: 15m5.938s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:88
Spec Goroutine
goroutine 81 [select]
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8caa8c8, 0xc001586820}, 0x81d9580?)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8ce5ea0, {0x8caa8c8, 0xc001586820}, 0x0)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x8caa938?, 0xc000704380?}, {{0x8cc2ba8?, 0xc000e26280?}, 0xc00093c648?}, 0xc0000d7768, {0xc00168e690, 0x14}, {0x81f1f5c, 0x11}, ...)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:694
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x81b78f8, 0x4}, ...}, ...)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:346
> github.com/Azure/ARO-HCP/test/e2e.init.func18.2({0x7f585c2cf198, 0xc000e74e10})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:89
|
| By("creating the HCP cluster")
> err = tc.CreateHCPClusterFromParam(
| ctx,
| GinkgoLogr,
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8cc2c80?, 0xc000e74e10?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 78
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96 @ 04/02/26 01:51:54.509
"ts"="2026-04-02 01:51:54.509787" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 01:51:54.509849" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-qwz4qp" "resourceGroup"="basic-cluster-448hmt"
"ts"="2026-04-02 01:51:54.642128" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-qwz4qp" "resourceGroup"=""
"ts"="2026-04-02 01:51:55.916040" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-8d7dc92573349c6d0052ecc8bc03042b1bfdfade27aae309024a834a40d01d5c.yaml.gz"
"ts"="2026-04-02 01:51:55.916147" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-8d7dc92573349c6d0052ecc8bc03042b1bfdfade27aae309024a834a40d01d5c.yaml.gz"
"ts"="2026-04-02 01:51:55.916233" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 01:51:55.917374" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-135"
"ts"="2026-04-02 01:51:55.917390" "level"=0 "msg"="collecting deployments" "resourceGroup"="basic-cluster-448hmt"
"ts"="2026-04-02 01:51:56.033502" "level"=0 "msg"="collecting operations" "resourceGroup"="basic-cluster-448hmt" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
"ts"="2026-04-02 01:51:56.066101" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-135" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 01:51:56.118598" "level"=0 "msg"="collecting operations" "resourceGroup"="basic-cluster-448hmt" "deployment"="customer-infra-basic-hcp-cluster-qwz4qp"
"ts"="2026-04-02 01:51:56.406662" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-135" "deployment"="msi-deployment-135"
"ts"="2026-04-02 01:51:56.541695" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 01:51:56.541805" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 01:51:56.541940" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="basic-cluster-448hmt"
Command Error: signal: killed
Deserializaion Error: no output from commandDeserializaion Error: no output from command Show raw failureRunning Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775093552 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should be able to use workload identity via the cluster OIDC issuer URL [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:169
STEP: creating a resource group @ 04/02/26 01:32:32.392
"ts"="2026-04-02 01:32:32.393025" "level"=0 "msg"="creating resource group" "resourceGroup"="oidc-wi-hwdf78"
STEP: creating cluster parameters @ 04/02/26 01:32:36.337
"ts"="2026-04-02 01:32:36.337842" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-04-02 01:32:36.337858" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources (infrastructure and managed identities) for cluster @ 04/02/26 01:32:36.337
STEP: creating the HCP cluster @ 04/02/26 01:33:48.487
"ts"="2026-04-02 01:33:48.487534" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="oidc-wi-cluster" "resourceGroup"="oidc-wi-hwdf78"
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216 @ 04/02/26 02:18:48.488
"ts"="2026-04-02 02:18:48.488819" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 02:18:48.488889" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-oidc-wi-cluster-qgwwhq" "resourceGroup"="oidc-wi-hwdf78"
"ts"="2026-04-02 02:18:48.616708" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-oidc-wi-cluster-qgwwhq" "resourceGroup"=""
"ts"="2026-04-02 02:18:50.026272" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-f2883517cea1430b4915f04acc052b067a0c30de1b37a57f443ec7d05cd4dc52.yaml.gz"
"ts"="2026-04-02 02:18:50.026377" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-f2883517cea1430b4915f04acc052b067a0c30de1b37a57f443ec7d05cd4dc52.yaml.gz"
"ts"="2026-04-02 02:18:50.026457" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 02:18:50.027629" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-90"
"ts"="2026-04-02 02:18:50.027651" "level"=0 "msg"="collecting deployments" "resourceGroup"="oidc-wi-hwdf78"
"ts"="2026-04-02 02:18:50.203926" "level"=0 "msg"="collecting operations" "resourceGroup"="oidc-wi-hwdf78" "deployment"="pooledNonMsiScopedAssignments-s6kwp7xg43dbi"
"ts"="2026-04-02 02:18:50.292891" "level"=0 "msg"="collecting operations" "resourceGroup"="oidc-wi-hwdf78" "deployment"="customer-infra-oidc-wi-cluster-qgwwhq"
"ts"="2026-04-02 02:18:50.386139" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-90" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 02:18:50.540807" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-90" "deployment"="msi-deployment-90"
"ts"="2026-04-02 02:18:50.676425" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 02:18:50.676568" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 02:18:50.676702" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="oidc-wi-hwdf78"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775093552 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should be able to use workload identity via the cluster OIDC issuer URL [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:169
STEP: creating a resource group @ 04/02/26 01:32:32.392
"ts"="2026-04-02 01:32:32.393025" "level"=0 "msg"="creating resource group" "resourceGroup"="oidc-wi-hwdf78"
STEP: creating cluster parameters @ 04/02/26 01:32:36.337
"ts"="2026-04-02 01:32:36.337842" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-04-02 01:32:36.337858" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources (infrastructure and managed identities) for cluster @ 04/02/26 01:32:36.337
STEP: creating the HCP cluster @ 04/02/26 01:33:48.487
"ts"="2026-04-02 01:33:48.487534" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="oidc-wi-cluster" "resourceGroup"="oidc-wi-hwdf78"
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216 @ 04/02/26 02:18:48.488
"ts"="2026-04-02 02:18:48.488819" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 02:18:48.488889" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-oidc-wi-cluster-qgwwhq" "resourceGroup"="oidc-wi-hwdf78"
"ts"="2026-04-02 02:18:48.616708" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-oidc-wi-cluster-qgwwhq" "resourceGroup"=""
"ts"="2026-04-02 02:18:50.026272" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-f2883517cea1430b4915f04acc052b067a0c30de1b37a57f443ec7d05cd4dc52.yaml.gz"
"ts"="2026-04-02 02:18:50.026377" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-f2883517cea1430b4915f04acc052b067a0c30de1b37a57f443ec7d05cd4dc52.yaml.gz"
"ts"="2026-04-02 02:18:50.026457" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 02:18:50.027629" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-90"
"ts"="2026-04-02 02:18:50.027651" "level"=0 "msg"="collecting deployments" "resourceGroup"="oidc-wi-hwdf78"
"ts"="2026-04-02 02:18:50.203926" "level"=0 "msg"="collecting operations" "resourceGroup"="oidc-wi-hwdf78" "deployment"="pooledNonMsiScopedAssignments-s6kwp7xg43dbi"
"ts"="2026-04-02 02:18:50.292891" "level"=0 "msg"="collecting operations" "resourceGroup"="oidc-wi-hwdf78" "deployment"="customer-infra-oidc-wi-cluster-qgwwhq"
"ts"="2026-04-02 02:18:50.386139" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-90" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 02:18:50.540807" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-90" "deployment"="msi-deployment-90"
"ts"="2026-04-02 02:18:50.676425" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 02:18:50.676568" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 02:18:50.676702" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="oidc-wi-hwdf78"
Command Error: signal: killed
Deserializaion Error: no output from commandtimeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
<*fmt.wrapErrors | 0xc000f32720>:
failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-n2k8wb, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-n2k8wb" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-n2k8wb, error: context deadline exceeded
{
msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-n2k8wb, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-n2k8wb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-n2k8wb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000e8e3a0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-n2k8wb",
},
<*fmt.wrapErrors | 0xc000f326f0>{
msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-n2k8wb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-n2k8wb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000e8e3a0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-n2k8wb",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
<*fmt.wrapErrors | 0xc000f32720>:
failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-n2k8wb, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-n2k8wb" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-n2k8wb, error: context deadline exceeded
{
msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-n2k8wb, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-n2k8wb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-n2k8wb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000e8e3a0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-n2k8wb",
},
<*fmt.wrapErrors | 0xc000f326f0>{
msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-n2k8wb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-n2k8wb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000e8e3a0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-n2k8wb",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredDeserializaion Error: no output from command Show raw failurenup.
Here's a current progress report:
Customer should be able to create an HCP cluster and manage pull secrets (Spec Runtime: 1h29m59.871s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:68
In [It] (Node Runtime: 1h29m59.871s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:68
At [By Step] Creating the cluster (Step Runtime: 13m55.176s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:126
Spec Goroutine
goroutine 106 [select]
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8caa8c8, 0xc0009b6370}, 0x81d9580?)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8ce5ea0, {0x8caa8c8, 0xc0009b6370}, 0x0)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x8caa938?, 0xc0006fe770?}, {{0x8cc2ba8?, 0xc0002fc000?}, 0xc0015b39f0?}, 0xc0000d7388, {0xc001462f00, 0x16}, {0x820df8d, 0x16}, ...)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:694
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x81b78f8, 0x4}, ...}, ...)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:346
> github.com/Azure/ARO-HCP/test/e2e.init.func13.2({0x7fe88dfb0c60, 0xc00094f710})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:127
|
| By("Creating the cluster")
> err = tc.CreateHCPClusterFromParam(ctx,
| GinkgoLogr,
| *resourceGroup.Name,
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8cc2c80?, 0xc00094f710?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 103
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133 @ 04/02/26 01:51:54.529
"ts"="2026-04-02 01:51:54.529471" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 01:51:54.529531" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-pullsecret-hcp-cluster-lx649m" "resourceGroup"="pullsecret-test-9fw4wg"
"ts"="2026-04-02 01:51:54.696878" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-pullsecret-hcp-cluster-lx649m" "resourceGroup"=""
"ts"="2026-04-02 01:51:56.149433" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-a760a96f7a26fb2915b8600ab91b283094b2c3bf2e64d6f8222da667d302697b.yaml.gz"
"ts"="2026-04-02 01:51:56.149565" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-a760a96f7a26fb2915b8600ab91b283094b2c3bf2e64d6f8222da667d302697b.yaml.gz"
"ts"="2026-04-02 01:51:56.149650" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 01:51:56.150896" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-144"
"ts"="2026-04-02 01:51:56.150922" "level"=0 "msg"="collecting deployments" "resourceGroup"="pullsecret-test-9fw4wg"
"ts"="2026-04-02 01:51:56.288868" "level"=0 "msg"="collecting operations" "resourceGroup"="pullsecret-test-9fw4wg" "deployment"="pooledNonMsiScopedAssignments-p2h32vyw2l2ma"
"ts"="2026-04-02 01:51:56.420385" "level"=0 "msg"="collecting operations" "resourceGroup"="pullsecret-test-9fw4wg" "deployment"="customer-infra-pullsecret-hcp-cluster-lx649m"
"ts"="2026-04-02 01:51:56.704432" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-144" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 01:51:56.829125" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-144" "deployment"="msi-deployment-144"
"ts"="2026-04-02 01:51:56.986545" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 01:51:56.986670" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 01:51:56.986793" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="pullsecret-test-9fw4wg"
Command Error: signal: killed
Deserializaion Error: no output from command
nup.
Here's a current progress report:
Customer should be able to create an HCP cluster and manage pull secrets (Spec Runtime: 1h29m59.871s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:68
In [It] (Node Runtime: 1h29m59.871s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:68
At [By Step] Creating the cluster (Step Runtime: 13m55.176s)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:126
Spec Goroutine
goroutine 106 [select]
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8caa8c8, 0xc0009b6370}, 0x81d9580?)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8ce5ea0, {0x8caa8c8, 0xc0009b6370}, 0x0)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x8caa938?, 0xc0006fe770?}, {{0x8cc2ba8?, 0xc0002fc000?}, 0xc0015b39f0?}, 0xc0000d7388, {0xc001462f00, 0x16}, {0x820df8d, 0x16}, ...)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:694
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x81b78f8, 0x4}, ...}, ...)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:346
> github.com/Azure/ARO-HCP/test/e2e.init.func13.2({0x7fe88dfb0c60, 0xc00094f710})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:127
|
| By("Creating the cluster")
> err = tc.CreateHCPClusterFromParam(ctx,
| GinkgoLogr,
| *resourceGroup.Name,
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8cc2c80?, 0xc00094f710?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 103
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133 @ 04/02/26 01:51:54.529
"ts"="2026-04-02 01:51:54.529471" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 01:51:54.529531" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-pullsecret-hcp-cluster-lx649m" "resourceGroup"="pullsecret-test-9fw4wg"
"ts"="2026-04-02 01:51:54.696878" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-pullsecret-hcp-cluster-lx649m" "resourceGroup"=""
"ts"="2026-04-02 01:51:56.149433" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-a760a96f7a26fb2915b8600ab91b283094b2c3bf2e64d6f8222da667d302697b.yaml.gz"
"ts"="2026-04-02 01:51:56.149565" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-a760a96f7a26fb2915b8600ab91b283094b2c3bf2e64d6f8222da667d302697b.yaml.gz"
"ts"="2026-04-02 01:51:56.149650" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 01:51:56.150896" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-144"
"ts"="2026-04-02 01:51:56.150922" "level"=0 "msg"="collecting deployments" "resourceGroup"="pullsecret-test-9fw4wg"
"ts"="2026-04-02 01:51:56.288868" "level"=0 "msg"="collecting operations" "resourceGroup"="pullsecret-test-9fw4wg" "deployment"="pooledNonMsiScopedAssignments-p2h32vyw2l2ma"
"ts"="2026-04-02 01:51:56.420385" "level"=0 "msg"="collecting operations" "resourceGroup"="pullsecret-test-9fw4wg" "deployment"="customer-infra-pullsecret-hcp-cluster-lx649m"
"ts"="2026-04-02 01:51:56.704432" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-144" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 01:51:56.829125" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-144" "deployment"="msi-deployment-144"
"ts"="2026-04-02 01:51:56.986545" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 01:51:56.986670" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 01:51:56.986793" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="pullsecret-test-9fw4wg"
Command Error: signal: killed
Deserializaion Error: no output from commandDeserializaion Error: no output from command Show raw failureRunning Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775093416 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:226
"ts"="2026-04-02 01:30:16.223704" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating resource group @ 04/02/26 01:30:16.592
"ts"="2026-04-02 01:30:16.592682" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-2tcvzl-6t9hp4"
STEP: creating cluster parameters at control plane version @ 04/02/26 01:30:20.35
"ts"="2026-04-02 01:30:20.350778" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-04-02 01:30:20.350792" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 04/02/26 01:30:20.35
STEP: creating the HCP cluster with version 4.21 @ 04/02/26 01:31:15.662
"ts"="2026-04-02 01:31:15.662325" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-2tcvzl" "resourceGroup"="rg-np-version-upgrade-2tcvzl-6t9hp4"
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132 @ 04/02/26 02:16:15.663
"ts"="2026-04-02 02:16:15.663428" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 02:16:15.663517" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-2tcvzl-zk272b" "resourceGroup"="rg-np-version-upgrade-2tcvzl-6t9hp4"
"ts"="2026-04-02 02:16:16.034813" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-2tcvzl-zk272b" "resourceGroup"=""
"ts"="2026-04-02 02:16:17.548875" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-04-02 02:16:17.548974" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-04-02 02:16:17.549030" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 02:16:17.550112" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-122"
"ts"="2026-04-02 02:16:17.550138" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-2tcvzl-6t9hp4"
"ts"="2026-04-02 02:16:17.698808" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-2tcvzl-6t9hp4" "deployment"="pooledNonMsiScopedAssignments-3rz66um7kjo3u"
"ts"="2026-04-02 02:16:17.791141" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-2tcvzl-6t9hp4" "deployment"="customer-infra-np-version-upgrade-cluster-2tcvzl-zk272b"
"ts"="2026-04-02 02:16:17.867337" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-122" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 02:16:18.022305" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-122" "deployment"="msi-deployment-122"
"ts"="2026-04-02 02:16:18.167675" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 02:16:18.167790" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 02:16:18.167929" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-2tcvzl-6t9hp4"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775093416 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:226
"ts"="2026-04-02 01:30:16.223704" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating resource group @ 04/02/26 01:30:16.592
"ts"="2026-04-02 01:30:16.592682" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-2tcvzl-6t9hp4"
STEP: creating cluster parameters at control plane version @ 04/02/26 01:30:20.35
"ts"="2026-04-02 01:30:20.350778" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-04-02 01:30:20.350792" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 04/02/26 01:30:20.35
STEP: creating the HCP cluster with version 4.21 @ 04/02/26 01:31:15.662
"ts"="2026-04-02 01:31:15.662325" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-2tcvzl" "resourceGroup"="rg-np-version-upgrade-2tcvzl-6t9hp4"
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132 @ 04/02/26 02:16:15.663
"ts"="2026-04-02 02:16:15.663428" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 02:16:15.663517" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-2tcvzl-zk272b" "resourceGroup"="rg-np-version-upgrade-2tcvzl-6t9hp4"
"ts"="2026-04-02 02:16:16.034813" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-2tcvzl-zk272b" "resourceGroup"=""
"ts"="2026-04-02 02:16:17.548875" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-04-02 02:16:17.548974" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-04-02 02:16:17.549030" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 02:16:17.550112" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-122"
"ts"="2026-04-02 02:16:17.550138" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-2tcvzl-6t9hp4"
"ts"="2026-04-02 02:16:17.698808" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-2tcvzl-6t9hp4" "deployment"="pooledNonMsiScopedAssignments-3rz66um7kjo3u"
"ts"="2026-04-02 02:16:17.791141" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-2tcvzl-6t9hp4" "deployment"="customer-infra-np-version-upgrade-cluster-2tcvzl-zk272b"
"ts"="2026-04-02 02:16:17.867337" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-122" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 02:16:18.022305" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-122" "deployment"="msi-deployment-122"
"ts"="2026-04-02 02:16:18.167675" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 02:16:18.167790" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 02:16:18.167929" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-2tcvzl-6t9hp4"
Command Error: signal: killed
Deserializaion Error: no output from commandtimeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
<*fmt.wrapErrors | 0xc0009e3290>:
failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-6jth45" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-6jth45, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-6jth45\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-6jth45, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000fcad80>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-6jth45",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
<*fmt.wrapErrors | 0xc0009e3290>:
failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-6jth45" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-6jth45, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-6jth45\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-6jth45, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000fcad80>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-6jth45",
},
<context.deadlineExceededError>{},
],
}
occurredtimeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
<*fmt.wrapErrors | 0xc0015f8f90>:
failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-j9ccbj, error: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-j9ccbj" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-j9ccbj, error: context deadline exceeded
{
msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-j9ccbj, error: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-j9ccbj\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-j9ccbj, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001634e80>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-j9ccbj",
},
<*fmt.wrapErrors | 0xc0015f8f60>{
msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-j9ccbj\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-j9ccbj, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001634e80>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-j9ccbj",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
<*fmt.wrapErrors | 0xc0015f8f90>:
failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-j9ccbj, error: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-j9ccbj" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-j9ccbj, error: context deadline exceeded
{
msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-j9ccbj, error: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-j9ccbj\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-j9ccbj, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001634e80>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-j9ccbj",
},
<*fmt.wrapErrors | 0xc0015f8f60>{
msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-j9ccbj\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-j9ccbj, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001634e80>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-j9ccbj",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredDeserializaion Error: no output from command Show raw failureRunning Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775093414 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should be able to update node pool labels and taints [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
STEP: creating a resource group @ 04/02/26 01:30:16.183
"ts"="2026-04-02 01:30:16.183334" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-labels-taints-kd6zb5"
STEP: creating cluster parameters @ 04/02/26 01:30:19.545
"ts"="2026-04-02 01:30:19.545044" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-04-02 01:30:19.545057" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 04/02/26 01:30:19.545
STEP: creating the HCP cluster @ 04/02/26 01:31:15.39
"ts"="2026-04-02 01:31:15.390262" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cluster-np-labels-taints" "resourceGroup"="rg-np-labels-taints-kd6zb5"
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91 @ 04/02/26 02:16:15.391
"ts"="2026-04-02 02:16:15.391756" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 02:16:15.391835" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-cv7855" "resourceGroup"="rg-np-labels-taints-kd6zb5"
"ts"="2026-04-02 02:16:15.522013" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-cv7855" "resourceGroup"=""
"ts"="2026-04-02 02:16:16.197911" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
"ts"="2026-04-02 02:16:16.198011" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
"ts"="2026-04-02 02:16:16.198082" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 02:16:16.199349" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-145"
"ts"="2026-04-02 02:16:16.199367" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-kd6zb5"
"ts"="2026-04-02 02:16:16.383278" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-145" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 02:16:16.517232" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-145" "deployment"="msi-deployment-145"
"ts"="2026-04-02 02:16:16.652118" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-kd6zb5" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k"
"ts"="2026-04-02 02:16:16.730191" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-kd6zb5" "deployment"="customer-infra-cluster-np-labels-taints-cv7855"
"ts"="2026-04-02 02:16:17.286302" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 02:16:17.286461" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 02:16:17.286613" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-kd6zb5"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1775093414 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should be able to update node pool labels and taints [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
STEP: creating a resource group @ 04/02/26 01:30:16.183
"ts"="2026-04-02 01:30:16.183334" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-labels-taints-kd6zb5"
STEP: creating cluster parameters @ 04/02/26 01:30:19.545
"ts"="2026-04-02 01:30:19.545044" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-04-02 01:30:19.545057" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 04/02/26 01:30:19.545
STEP: creating the HCP cluster @ 04/02/26 01:31:15.39
"ts"="2026-04-02 01:31:15.390262" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cluster-np-labels-taints" "resourceGroup"="rg-np-labels-taints-kd6zb5"
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91 @ 04/02/26 02:16:15.391
"ts"="2026-04-02 02:16:15.391756" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-02 02:16:15.391835" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-cv7855" "resourceGroup"="rg-np-labels-taints-kd6zb5"
"ts"="2026-04-02 02:16:15.522013" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-cv7855" "resourceGroup"=""
"ts"="2026-04-02 02:16:16.197911" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
"ts"="2026-04-02 02:16:16.198011" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
"ts"="2026-04-02 02:16:16.198082" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-02 02:16:16.199349" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-145"
"ts"="2026-04-02 02:16:16.199367" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-kd6zb5"
"ts"="2026-04-02 02:16:16.383278" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-145" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-02 02:16:16.517232" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-145" "deployment"="msi-deployment-145"
"ts"="2026-04-02 02:16:16.652118" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-kd6zb5" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k"
"ts"="2026-04-02 02:16:16.730191" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-kd6zb5" "deployment"="customer-infra-cluster-np-labels-taints-cv7855"
"ts"="2026-04-02 02:16:17.286302" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-02 02:16:17.286461" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-02 02:16:17.286613" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-kd6zb5"
Command Error: signal: killed
Deserializaion Error: no output from commandunknown (1)Process did not finish before 2h0m0s timeout Show raw failure{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-02T02:21:49Z"} |