No runs match your search.
Environment: DEV
| Time (UTC) | Runtime | Job | Region | Failed at | Result | PR | Failed tests | Details |
|---|---|---|---|---|---|---|---|---|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | e2e | failed | #4652 (merged) | 1 | e2e (1)VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4 Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:171]: Timed out after 2700.006s.
Expected success, but got an error:
<*errors.joinError | 0xc000114e88>:
VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0, targetMinor=4.21.0) failed: clusterversion status.history has no version in target minor "4.21.0"
{
errs: [
<*fmt.wrapError | 0xc000a1a7a0>{
msg: "VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0, targetMinor=4.21.0) failed: clusterversion status.history has no version in target minor \"4.21.0\"",
err: <*errors.errorString | 0xc0007aa580>{
s: "clusterversion status.history has no version in target minor \"4.21.0\"",
},
},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:171]: Timed out after 2700.006s.
Expected success, but got an error:
<*errors.joinError | 0xc000114e88>:
VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0, targetMinor=4.21.0) failed: clusterversion status.history has no version in target minor "4.21.0"
{
errs: [
<*fmt.wrapError | 0xc000a1a7a0>{
msg: "VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0, targetMinor=4.21.0) failed: clusterversion status.history has no version in target minor \"4.21.0\"",
err: <*errors.errorString | 0xc0007aa580>{
s: "clusterversion status.history has no version in target minor \"4.21.0\"",
},
},
],
} |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | e2e | failed | #4640 (merged) | 1 | e2e (1)read access verification failed: get pod <pod> kube-system: pods is forbidden: User "system:sre-break-glass:<uuid>" cannot list resource "pods" in API group "" in the namespace "<namespace>" Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:281]: Timed out after 30.001s.
Expected success, but got an error:
<*fmt.wrapError | 0xc001030540>:
read access verification failed: get pods -n kube-system: pods is forbidden: User "system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652" cannot list resource "pods" in API group "" in the namespace "kube-system"
get configmaps -n kube-system: configmaps is forbidden: User "system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
{
msg: "read access verification failed: get pods -n kube-system: pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"\nget configmaps -n kube-system: configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.joinError | 0xc000fb1de8>{
errs: [
<*fmt.wrapError | 0xc001100b60>{
msg: "get pods -n kube-system: pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.StatusError | 0xc001106b40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
Reason: "Forbidden",
Details: {Name: "", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
Code: 403,
},
},
},
<*fmt.wrapError | 0xc0010304e0>{
msg: "get configmaps -n kube-system: configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.StatusError | 0xc001020780>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
Reason: "Forbidden",
Details: {Name: "", Group: "", Kind: "configmaps", UID: "", Causes: nil, RetryAfterSeconds: 0},
Code: 403,
},
},
},
],
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:281]: Timed out after 30.001s.
Expected success, but got an error:
<*fmt.wrapError | 0xc001030540>:
read access verification failed: get pods -n kube-system: pods is forbidden: User "system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652" cannot list resource "pods" in API group "" in the namespace "kube-system"
get configmaps -n kube-system: configmaps is forbidden: User "system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
{
msg: "read access verification failed: get pods -n kube-system: pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"\nget configmaps -n kube-system: configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.joinError | 0xc000fb1de8>{
errs: [
<*fmt.wrapError | 0xc001100b60>{
msg: "get pods -n kube-system: pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.StatusError | 0xc001106b40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
Reason: "Forbidden",
Details: {Name: "", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
Code: 403,
},
},
},
<*fmt.wrapError | 0xc0010304e0>{
msg: "get configmaps -n kube-system: configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
err: <*errors.StatusError | 0xc001020780>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
},
Status: "Failure",
Message: "configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
Reason: "Forbidden",
Details: {Name: "", Group: "", Kind: "configmaps", UID: "", Causes: nil, RetryAfterSeconds: 0},
Code: 403,
},
},
},
],
},
} |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | e2e | failed | #4607 (merged) | 2 | e2e (2)ERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.; provider Microsoft.Network Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
<*fmt.wrapError | 0xc000757760>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-subnet-reuse-cjvhd4" in resourcegroup="rg-cluster-nsg-subnet-reuse-nsb4p6" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-cjvhd4/operationStatuses/08584270527184429378
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-subnet-reuse-cjvhd4\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-nsb4p6\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-cjvhd4/operationStatuses/08584270527184429378\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000757740>{
msg: "failed waiting for deployment \"customer-infra-cluster-subnet-reuse-cjvhd4\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-nsb4p6\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-cjvhd4/operationStatuses/08584270527184429378\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0008e8cc0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
<*fmt.wrapError | 0xc000757760>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-subnet-reuse-cjvhd4" in resourcegroup="rg-cluster-nsg-subnet-reuse-nsb4p6" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-cjvhd4/operationStatuses/08584270527184429378
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-subnet-reuse-cjvhd4\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-nsb4p6\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-cjvhd4/operationStatuses/08584270527184429378\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000757740>{
msg: "failed waiting for deployment \"customer-infra-cluster-subnet-reuse-cjvhd4\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-nsb4p6\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-cjvhd4/operationStatuses/08584270527184429378\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0008e8cc0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.; provider Microsoft.Network Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
<*fmt.wrapError | 0xc0014d4940>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-2x54nn" in resourcegroup="customer-rg-h9bxs7" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-h9bxs7/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-2x54nn/operationStatuses/08584270525680151780
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-2x54nn\" in resourcegroup=\"customer-rg-h9bxs7\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-h9bxs7/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-2x54nn/operationStatuses/08584270525680151780\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0014d4920>{
msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-2x54nn\" in resourcegroup=\"customer-rg-h9bxs7\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-h9bxs7/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-2x54nn/operationStatuses/08584270525680151780\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000d84cf0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
<*fmt.wrapError | 0xc0014d4940>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-2x54nn" in resourcegroup="customer-rg-h9bxs7" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-h9bxs7/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-2x54nn/operationStatuses/08584270525680151780
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-2x54nn\" in resourcegroup=\"customer-rg-h9bxs7\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-h9bxs7/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-2x54nn/operationStatuses/08584270525680151780\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0014d4920>{
msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-2x54nn\" in resourcegroup=\"customer-rg-h9bxs7\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-h9bxs7/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-2x54nn/operationStatuses/08584270525680151780\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000d84cf0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4625 (merged) | 1 | provision (1)Internal error occurred: failed calling webhook "validation.istio.io": failed to call webhook: Post "<url>": proxy error from localhost:9443 while dialing <ip>:<port>, code 500: 500 Internal Server Error Show raw failuretime=2026-03-26T19:27:42.856Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-26T19:27:43.551Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:571b690d-4bcd-4d64-8d4e-a350c2693d67 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j3442048] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j3442048.hcpsvc.osadev.cloud] image:map[digest:sha256:7b5ae3d95bc764234f0b78cbc111ee7528b6efb8b2a9027e6e6a707fb66cb5b9 registry:arohcpsvcdev.azurecr.io repository:ci-op-br33ciqz/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-br33ciqz/pipeline:pull workloadIdentityClientId:442e39d9-a6b5-4cab-9f5d-38d223e0b76e workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:a3867303-90c6-46e9-b871-b2d7f79da93a workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]" time=2026-03-26T19:27:43.551Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-26T19:27:43.819Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy namespace=sessiongate time=2026-03-26T19:27:44.123Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy dryRun=false time=2026-03-26T19:27:44.123Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-26T19:27:44.236Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-26T19:27:51.065Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.6:15017, code 500: 500 Internal Server Error" time=2026-03-26T19:27:51.065Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-26T19:27:51.065Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-26T19:27:52.732Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy release=sessiongate namespace=sessiongate status=failed description="Release \"sessiongate\" failed: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.6:15017, code 500: 500 Internal Server Error" values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:571b690d-4bcd-4d64-8d4e-a350c2693d67 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j3442048] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j3442048.hcpsvc.osadev.cloud] image:map[digest:sha256:7b5ae3d95bc764234f0b78cbc111ee7528b6efb8b2a9027e6e6a707fb66cb5b9 registry:arohcpsvcdev.azurecr.io repository:ci-op-br33ciqz/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-br33ciqz/pipeline:pull workloadIdentityClientId:442e39d9-a6b5-4cab-9f5d-38d223e0b76e workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:a3867303-90c6-46e9-b871-b2d7f79da93a workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]" time=2026-03-26T19:27:52.735Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5SUQU/cPhDF7/kUVk6stI5g//ypSsWBQqVKlaoVcKLqwXEeicHxpOMxCMSHr0LYEoVlF27OZOY3z+OnyTxEMSIltojqSFVGjJjSY4fKa1j54UJ1GIVdqOdqCP00Lf6FcIvwFImdsavw7FemlFL5OfjWWRxbSylIPld5RIyOQm0Erz6HmhOfooDPyE8z9PNZtyaYGi0G5KpwXYWlIEzeg9/otrlIe5gKrOFhxVHYAPnqQuVCvYFVrs34GGOiZwvzFJ2n+9Wcto/+zXxtweKunO3PjCtGbIaRmpuoXRRH2oWaEeOKdWx5mbwfXalL3m8RfA7LkCXTravAJ970vIkHbLex70mKQu3Zs6NPceWCmzxdLEbAwjDpxnaFeUiMwlI79tSzgd83vuMkDbF7MH2/JXln75+kek93U7EfBwSS5u3Jfb+4WJ5RWqNtvdvwJyFK3xBB+nd97W7NQ5I2SZopJ/v9JbtJJThAEL/1SyBmj+quAUOJaxHFtJ0qIXdAUDuVEfThnXyxuzjQu//pxcHF3ufDxafD/f3LfKaKQm3I+X/vMp/Nskd1TS6oGxeqIxcCeLS6KKiXjTVeVdMdlT2qjqn/+yJ0rjqqhuT3QebKUz1XLWI0dY8krsCqvB9dvkK0fwMAAP//DSzs0mAFAAA=" time=2026-03-26T19:27:52.737Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-26T19:27:52.737Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.6:15017, code 500: 500 Internal Server Error" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4653 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | e2e | failed | #4652 (merged) | 3 | e2e (3)ERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.; provider Microsoft.Network Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
<*fmt.wrapError | 0xc000eae9e0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-subnet-reuse-n58pg4" in resourcegroup="rg-cluster-nsg-subnet-reuse-wcrvrb" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-n58pg4/operationStatuses/08584270553481553506
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-subnet-reuse-n58pg4\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-wcrvrb\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-n58pg4/operationStatuses/08584270553481553506\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000eae9c0>{
msg: "failed waiting for deployment \"customer-infra-cluster-subnet-reuse-n58pg4\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-wcrvrb\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-n58pg4/operationStatuses/08584270553481553506\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0009d2210>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
<*fmt.wrapError | 0xc000eae9e0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-subnet-reuse-n58pg4" in resourcegroup="rg-cluster-nsg-subnet-reuse-wcrvrb" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-n58pg4/operationStatuses/08584270553481553506
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-subnet-reuse-n58pg4\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-wcrvrb\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-n58pg4/operationStatuses/08584270553481553506\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000eae9c0>{
msg: "failed waiting for deployment \"customer-infra-cluster-subnet-reuse-n58pg4\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-wcrvrb\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-n58pg4/operationStatuses/08584270553481553506\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0009d2210>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.; provider Microsoft.Network Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
<*fmt.wrapError | 0xc000046020>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-qdjsd4" in resourcegroup="customer-rg-nqljpg" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-nqljpg/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-qdjsd4/operationStatuses/08584270551842672141
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-qdjsd4\" in resourcegroup=\"customer-rg-nqljpg\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-nqljpg/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-qdjsd4/operationStatuses/08584270551842672141\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000046000>{
msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-qdjsd4\" in resourcegroup=\"customer-rg-nqljpg\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-nqljpg/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-qdjsd4/operationStatuses/08584270551842672141\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0010ba450>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
<*fmt.wrapError | 0xc000046020>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-qdjsd4" in resourcegroup="customer-rg-nqljpg" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-nqljpg/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-qdjsd4/operationStatuses/08584270551842672141
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-qdjsd4\" in resourcegroup=\"customer-rg-nqljpg\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-nqljpg/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-qdjsd4/operationStatuses/08584270551842672141\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc000046000>{
msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-qdjsd4\" in resourcegroup=\"customer-rg-nqljpg\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-nqljpg/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-qdjsd4/operationStatuses/08584270551842672141\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc0010ba450>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:164]: Unexpected error:
<*exported.ResponseError | 0xc0008f9ad0>:
PATCH https://rp.j3624320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-5pq6sw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-d4srhh
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Required value: field cannot be null",
"target": "properties.platform.vnetIntegrationSubnetId"
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Request-Id": [
"7a22645f-db58-4a17-b26f-09787f0217a0",
],
"Date": [
"Thu, 26 Mar 2026 19:36:23 GMT",
],
"Content-Length": ["188"],
"X-Envoy-Upstream-Service-Time": ["65"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000ca4ec0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 82, 101, 113, 117, 105, 114, 101, 100, 32, 118, 97, 108, 117, 101, 58, 32, 102, 105, 101, 108, 100, 32, 99, 97, 110, 110, 111, 116, 32, 98, 101, 32, 110, 117, 108, 108, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 46, 112, 108, 97, 116, 102, 111, 114, 109, 46, 118, 110, 101, 116, 73, 110, 116, 101, 103, 114, 97, 116, 105, 111, 110, 83, 117, 98, 110, 101, 116, 73, 100, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 188,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PATCH",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j3624320.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-5pq6sw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-d4srhh",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2025-12-23-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"Authorization": ["redacted"],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Accept": ["application/json"],
"Content-Length": ["66"],
"Content-Type": ["application/json"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-26T19:36:23Z\"}",
],
},
Body: <*http.readTrackingBody | 0xc000c8a7f8>{
ReadCloser: <*runtime.retryableRequestBody | 0xc0007e9380>{
body: <exported.nopCloser>{
ReadSeeker: <*bytes.Reader | 0xc0004f7050>{
s: "{\"properties\":{\"version\":{\"channelGroup\":\"stable\",\"id\":\"4.21.6\"}}}",
i: 66,
prevRune: -1,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:164]: Unexpected error:
<*exported.ResponseError | 0xc0008f9ad0>:
PATCH https://rp.j3624320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-5pq6sw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-d4srhh
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Required value: field cannot be null",
"target": "properties.platform.vnetIntegrationSubnetId"
}
}
--------------------------------------------------------------------------------
{
ErrorCode: "InvalidRequestContent",
StatusCode: 400,
RawResponse: {
Status: "400 Bad Request",
StatusCode: 400,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor: 0,
Header: {
"X-Ms-Request-Id": [
"7a22645f-db58-4a17-b26f-09787f0217a0",
],
"Date": [
"Thu, 26 Mar 2026 19:36:23 GMT",
],
"Content-Length": ["188"],
"X-Envoy-Upstream-Service-Time": ["65"],
"Server": ["istio-envoy"],
"Content-Type": ["application/json"],
"X-Ms-Error-Code": [
"InvalidRequestContent",
],
},
Body: <*exported.nopClosingBytesReader | 0xc000ca4ec0>{
s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 82, 101, 113, 117, 105, 114, 101, 100, 32, 118, 97, 108, 117, 101, 58, 32, 102, 105, 101, 108, 100, 32, 99, 97, 110, 110, 111, 116, 32, 98, 101, 32, 110, 117, 108, 108, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 46, 112, 108, 97, 116, 102, 111, 114, 109, 46, 118, 110, 101, 116, 73, 110, 116, 101, 103, 114, 97, 116, 105, 111, 110, 83, 117, 98, 110, 101, 116, 73, 100, 34, 10, 32, 32, 32, 32, 125, 10, 125],
i: 0,
},
ContentLength: 188,
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: {
Method: "PATCH",
URL: {
Scheme: "https",
Opaque: "",
User: nil,
Host: "rp.j3624320.hcpsvc.osadev.cloud",
Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-5pq6sw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-d4srhh",
RawPath: "",
OmitHost: false,
ForceQuery: false,
RawQuery: "api-version=2025-12-23-preview",
Fragment: "",
RawFragment: "",
},
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: {
"Authorization": ["redacted"],
"X-Ms-Identity-Url": [
"https://dummyhost.identity.azure.net",
],
"Accept": ["application/json"],
"Content-Length": ["66"],
"Content-Type": ["application/json"],
"User-Agent": [
"azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
],
"X-Ms-Arm-Resource-System-Data": [
"{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-26T19:36:23Z\"}",
],
},
Body: <*http.readTrackingBody | 0xc000c8a7f8>{
ReadCloser: <*runtime.retryableRequestBody | 0xc0007e9380>{
body: <exported.nopCloser>{
ReadSeeker: <*bytes.Reader | 0xc0004f7050>{
s: "{\"properties\":{\"version\":{\"channelGroup\":\"stable\",\"id\":\"4.21.6\"}}}",
i: 66,
prevRune: -1,
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | provision | failed | #4638 (closed) | 1 | provision (1)ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid Show raw failuretime=2026-03-26T18:16:27.487Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-26T18:16:30.165Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-26T18:16:31.296Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=6370d42ed2cab467b9d8a9e641c179817ed19b04da53274e67464cff44bd32f4
time=2026-03-26T18:18:01.839Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j4555904/providers/Microsoft.Resources/deployments/6370d42ed2cab467b9d8a9e641c179817ed19b04da53274e67464cff44bd32f4/operationStatuses/08584270578948601026\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4555904/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4555904/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j4555904\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Internal error\\\",\\r\\n \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 3e5d19b8-bdfb-4823-b567-c6d7077f42fb:3/26/2026 6:17:48 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4004 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4507 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4578 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4507 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4607 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4507 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4630 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4638 (closed) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4651 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4507 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4645 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4507 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | e2e | failed | #4640 (merged) | 1 | e2e (1)node pool upgrade verification failed: <node> (version 4 Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.003s.
Expected success, but got an error:
<*errors.errorString | 0xc00090ba80>:
node pool upgrade verification failed: q0f3h5j5u6h0w3l-npupgrade-4-20-kdc5m-k9xcn (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"q0f3h5j5u6h0w3l-npupgrade-4-20-bgvp5-gwb6f","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"q0f3h5j5u6h0w3l-npupgrade-4-20-bgvp5-zf4gf","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"q0f3h5j5u6h0w3l-npupgrade-4-20-kdc5m-k9xcn","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a9f74fb0da7fb9f7d70c01953cb70fdda6a22bb9182bc5358f5fd6d957f4f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef42cb417552fe1e38eedc30e92f55c8c50aae1aa1b0b7094ab56118efe0d96c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5dc767cd6267e006720af8cef541e23bc64d94ef9b1bf3a8b44b1fda5d9d8871","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dee9b6866db30456c277ff1f7a114dd4ce25bff8aced70218623af9423ec3cf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3c906f4908c692697ae58891ef463ee62a23d4356bf00786947cc1dc02e4d53","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78c23f5bccb6e82ade8428c8886ddc0291faec496590b2e6b5375ac6c0476dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ff94628410a702de0cb3e1a715917829b36c3def524dcc0850ac34a811f8f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:876fb57ffb255ed75a452405bcd1d116788a34b19264886e6e087ff544c17083","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ff360b712000d39cbb58158186ff0a18764278e6fa98ef9f21d113b978d9a79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77d8ddf2957025a39616328b5cf2b012de6cedce63fb79eebc5270d7b3401574","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec94607aa5fce47591b1ed1939e86a0e1c43cc8ff439f42662c9ed32e414595e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2962c6c31d7078b34cf7ba0c9b0ea685dc539cea67b06bd38b6ac566298bb790","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:321163a14934467a86e358bbb3115fbe104ad814bcaf7423f259eb82150da703","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7376a666810d810549b4e270d9a2fd6bcfd8799370ff52476a4f799cd4e287d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f426776d3ef70b585018b8d813f5eedd70373536681c6240df224ebf9ff6329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2906e0869191da12130ed79a57221aae735ef026d0f909534c7cbfbca131128","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3d68a9523f4c5c22a48dfa151bbfde78b2f360697afe3b9d8bfb53e8c359bf95","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7983b528aa73931fdaa817858ecd416dea0ec5a6917bebf5bd11b4eb83c752c9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f13a1e673c882ee926e97270249a20d3c40797a8c7c5abb2fda9c98c24a2eb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d4067116d68c1aabb5ee21c4addce05f6849f104d7a7f87f5f1fcef7990a3aee","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2af652f68b17ed49c2565928e4c101880e949c4484c6ee958793a8e668e47bea","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e199029f436d7247195d0c5b22909cb874ac67bd87d84c597a83ef7bd0d87ab0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:41e0dcd2f91df9234249f168b3fbe6d55cfdb3ebd2e598d5edfa47f905c046cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9998ff90e081aac492498314bd02225cc3a8636d8c714ce8b618f4ee1de28c21","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a054cfe1b8330bd57d762921ca5cd281ad2fdd7994fd3093ffadb5d767a564","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b369532d212af90d1232bcf0a84bc2616106733f4eae8d4aaf69fad595e1d67"]}]
{
s: "node pool upgrade verification failed: q0f3h5j5u6h0w3l-npupgrade-4-20-kdc5m-k9xcn (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"q0f3h5j5u6h0w3l-npupgrade-4-20-bgvp5-gwb6f\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9\",\"arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda\",\"arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.003s.
Expected success, but got an error:
<*errors.errorString | 0xc00090ba80>:
node pool upgrade verification failed: q0f3h5j5u6h0w3l-npupgrade-4-20-kdc5m-k9xcn (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"q0f3h5j5u6h0w3l-npupgrade-4-20-bgvp5-gwb6f","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"q0f3h5j5u6h0w3l-npupgrade-4-20-bgvp5-zf4gf","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"q0f3h5j5u6h0w3l-npupgrade-4-20-kdc5m-k9xcn","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a9f74fb0da7fb9f7d70c01953cb70fdda6a22bb9182bc5358f5fd6d957f4f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef42cb417552fe1e38eedc30e92f55c8c50aae1aa1b0b7094ab56118efe0d96c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5dc767cd6267e006720af8cef541e23bc64d94ef9b1bf3a8b44b1fda5d9d8871","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dee9b6866db30456c277ff1f7a114dd4ce25bff8aced70218623af9423ec3cf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3c906f4908c692697ae58891ef463ee62a23d4356bf00786947cc1dc02e4d53","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78c23f5bccb6e82ade8428c8886ddc0291faec496590b2e6b5375ac6c0476dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ff94628410a702de0cb3e1a715917829b36c3def524dcc0850ac34a811f8f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:876fb57ffb255ed75a452405bcd1d116788a34b19264886e6e087ff544c17083","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ff360b712000d39cbb58158186ff0a18764278e6fa98ef9f21d113b978d9a79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77d8ddf2957025a39616328b5cf2b012de6cedce63fb79eebc5270d7b3401574","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec94607aa5fce47591b1ed1939e86a0e1c43cc8ff439f42662c9ed32e414595e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2962c6c31d7078b34cf7ba0c9b0ea685dc539cea67b06bd38b6ac566298bb790","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:321163a14934467a86e358bbb3115fbe104ad814bcaf7423f259eb82150da703","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7376a666810d810549b4e270d9a2fd6bcfd8799370ff52476a4f799cd4e287d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f426776d3ef70b585018b8d813f5eedd70373536681c6240df224ebf9ff6329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2906e0869191da12130ed79a57221aae735ef026d0f909534c7cbfbca131128","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3d68a9523f4c5c22a48dfa151bbfde78b2f360697afe3b9d8bfb53e8c359bf95","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7983b528aa73931fdaa817858ecd416dea0ec5a6917bebf5bd11b4eb83c752c9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f13a1e673c882ee926e97270249a20d3c40797a8c7c5abb2fda9c98c24a2eb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d4067116d68c1aabb5ee21c4addce05f6849f104d7a7f87f5f1fcef7990a3aee","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2af652f68b17ed49c2565928e4c101880e949c4484c6ee958793a8e668e47bea","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e199029f436d7247195d0c5b22909cb874ac67bd87d84c597a83ef7bd0d87ab0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:41e0dcd2f91df9234249f168b3fbe6d55cfdb3ebd2e598d5edfa47f905c046cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9998ff90e081aac492498314bd02225cc3a8636d8c714ce8b618f4ee1de28c21","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a054cfe1b8330bd57d762921ca5cd281ad2fdd7994fd3093ffadb5d767a564","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b369532d212af90d1232bcf0a84bc2616106733f4eae8d4aaf69fad595e1d67"]}]
{
s: "node pool upgrade verification failed: q0f3h5j5u6h0w3l-npupgrade-4-20-kdc5m-k9xcn (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"q0f3h5j5u6h0w3l-npupgrade-4-20-bgvp5-gwb6f\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9\",\"arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda\",\"arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4645 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel batch |
n/a | n/a | passed | #4510 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4485 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel batch |
n/a | e2e | failed | #4485 (merged) | 1 | e2e (1)timeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
<*fmt.wrapErrors | 0xc001444f90>:
failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-2nbkt4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded
{
msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-2nbkt4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0009456e0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4",
},
<*fmt.wrapErrors | 0xc001444f60>{
msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-2nbkt4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0009456e0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
<*fmt.wrapErrors | 0xc001444f90>:
failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-2nbkt4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded
{
msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-2nbkt4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0009456e0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4",
},
<*fmt.wrapErrors | 0xc001444f60>{
msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-2nbkt4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0009456e0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel batch |
n/a | n/a | passed | #4485 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4648 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4577 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4647 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | e2e | failed | #4473 (merged) | 1 | e2e (1)timeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
<*fmt.wrapErrors | 0xc0011d91a0>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-48kgkn" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-48kgkn\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001196aa0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
<*fmt.wrapErrors | 0xc0011d91a0>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-48kgkn" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-48kgkn\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc001196aa0>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn",
},
<context.deadlineExceededError>{},
],
}
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4510 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel batch |
n/a | n/a | passed | #4505 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4615 (closed) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4645 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4485 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4605 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel batch |
n/a | provision | failed | #4505 (merged) | 1 | provision (1)Internal error occurred: failed calling webhook "validation.istio.io": failed to call webhook: Post "<url>": EOF Show raw failuretime=2026-03-26T10:48:23.528Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-26T10:48:24.177Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:2d906c8b-39a3-4047-8d40-cb975605ba4a cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j8294784] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j8294784.hcpsvc.osadev.cloud] image:map[digest:sha256:50b5fb5bf5e6af13c266dff43d6597d4e81da69b3fcc69a8884876fab568ea81 registry:arohcpsvcdev.azurecr.io repository:ci-op-6wcrgjtb/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-6wcrgjtb/pipeline:pull workloadIdentityClientId:277dd18f-32d4-4cb3-9d0f-7a85c95829b6 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:3301eb93-f945-4afd-b88e-96b8cc9739ba workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]" time=2026-03-26T10:48:24.177Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-26T10:48:24.382Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy namespace=sessiongate time=2026-03-26T10:48:24.693Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy dryRun=false time=2026-03-26T10:48:24.694Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-26T10:48:24.820Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-26T10:48:32.557Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": EOF" time=2026-03-26T10:48:32.557Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-26T10:48:32.557Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-26T10:48:34.078Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy release=sessiongate namespace=sessiongate status=failed description="Release \"sessiongate\" failed: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": EOF" values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:2d906c8b-39a3-4047-8d40-cb975605ba4a cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j8294784] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j8294784.hcpsvc.osadev.cloud] image:map[digest:sha256:50b5fb5bf5e6af13c266dff43d6597d4e81da69b3fcc69a8884876fab568ea81 registry:arohcpsvcdev.azurecr.io repository:ci-op-6wcrgjtb/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-6wcrgjtb/pipeline:pull workloadIdentityClientId:277dd18f-32d4-4cb3-9d0f-7a85c95829b6 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:3301eb93-f945-4afd-b88e-96b8cc9739ba workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]" time=2026-03-26T10:48:34.081Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6RUQWvbTBS861csOsXgFf6cED5cckiTQqFQjOtTSg+r1VjeeLVPvLeycfCPL4rtRji249Lb6mlmtPM0TOIRFUOoYQtRd6ow0USTe1xR/gwbv7lQjCSyC2VfbUffTYU/IywRXidSG7sf934mSimV/gAvncW9tdSEmPZVKhBxFEoT8e5xy3nwjUTwhPwhQu/OujLBlKiwldwTW8ZnFwoXykOipRCZvAfr/Cji7zQ8TAHW8LCxvc95zd0SLnP/iNrTem/tH/DagqObOdueGTOGzMEtyCxEO4mOtAslQ2Svdd/EObF7Ma2pMXln169472l1yDh6mbMCFSI7K1o+NnVWJ1Ccn1721+l0PKHmiPTxsDUSqZrs8v+ImQuu/WgHLlmHlxkmPbd1Zl4aRmapOkzg6dicidzlWTu1McvjxvtOdOvG+w+DaRlxzLR0BfjBm3c/Voutj0cm+fUpWTQ5OCBCvrQlIMlGreZgqOgqSDRVrXLEFRDUVWEi2vFVOhwMb/XgWg9vp/8NRjf/j4Y3T2lPZZk6g7kePqW9XrJRz+SCWrhQ3LkQwJ3qoqDeGqtbVYcdlWxUzdS+fbtoX9VUbMGXifSVp7KvKoiYspUkLsAqX3fMFxD7OwAA//8kk60oYAUAAA==" time=2026-03-26T10:48:34.082Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy time=2026-03-26T10:48:34.083Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": EOF" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4644 (closed) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4640 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4643 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | e2e | failed | #4577 (merged) | 1 | e2e (1)CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124]: Unexpected error:
<*fmt.wrapErrors | 0xc000d048d0>:
failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b, error: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-vsdn4b" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b, error: failed waiting for nodepool=\"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vsdn4b\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0002c7a30>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b",
},
<*fmt.wrapError | 0xc00145e300>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vsdn4b\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124]: Unexpected error:
<*fmt.wrapErrors | 0xc000d048d0>:
failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b, error: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-vsdn4b" to finish creating: context deadline exceeded
{
msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b, error: failed waiting for nodepool=\"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vsdn4b\" to finish creating: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0002c7a30>{
s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b",
},
<*fmt.wrapError | 0xc00145e300>{
msg: "failed waiting for nodepool=\"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vsdn4b\" to finish creating: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
],
}
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4642 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | provision | failed | #4446 (merged) | 1 | provision (1)resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress context deadline exceeded Show raw failuretime=2026-03-26T09:48:18.147Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-26T09:48:18.853Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[armPermissionsManagerIdentity:map[certName:armHelperCert2 clientId:3331e670-0804-48e8-a086-6241671ddc93 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj1912448.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj1912448.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clusterScopedIdentitiesConfig:map[roleSetName:dev] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:e9330836-104c-4ff6-8896-5e80a6561853 databaseName:arohcpprow-rp-j1912448 databaseUrl:https://arohcpprow-rp-j1912448.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-icg9x6nq/pipeline@sha256:d7affa8c94e1be1641298f474a1a1409323fb682a03fba66b2735fdfe48a905a replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-icg9x6nq/pipeline:pull workloadIdentityClientId:bbe90c53-94c0-41a0-a862-22f9c8836d86 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:e9330836-104c-4ff6-8896-5e80a6561853 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-03-26T09:48:18.853Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-26T09:48:18.912Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp time=2026-03-26T09:48:19.026Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false time=2026-03-26T09:48:19.026Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-26T09:48:19.137Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-26T09:53:23.504Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" time=2026-03-26T09:53:23.504Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-26T09:53:23.504Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-26T09:53:27.747Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy release=aro-hcp-backend-dev namespace=aro-hcp status=failed description="Release \"aro-hcp-backend-dev\" failed: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[armPermissionsManagerIdentity:map[certName:armHelperCert2 clientId:3331e670-0804-48e8-a086-6241671ddc93 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj1912448.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj1912448.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clusterScopedIdentitiesConfig:map[roleSetName:dev] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:e9330836-104c-4ff6-8896-5e80a6561853 databaseName:arohcpprow-rp-j1912448 databaseUrl:https://arohcpprow-rp-j1912448.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-icg9x6nq/pipeline@sha256:d7affa8c94e1be1641298f474a1a1409323fb682a03fba66b2735fdfe48a905a replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-icg9x6nq/pipeline:pull workloadIdentityClientId:bbe90c53-94c0-41a0-a862-22f9c8836d86 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:e9330836-104c-4ff6-8896-5e80a6561853 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-03-26T09:53:27.748Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/4xTTW8aMRS876942hNI64hCGzVUOaRpT1Uq1PSUqgevPV0cvH6rZy+IiB9fLSRhhSDhOm9m3oc9mUciQeRWDCJdk9VJJ116DLh8hEk/XLDTmMSFqqAd9FPXeIWwRNgisdHmBR7+yYiI8lsO/1x1p5u8oLzUZoFglX5qBUrakFwNZbaUrq6F1dw0efGG9iT7F3t0sIe2EAUPkxyHE8yvLlgXqr7z+8JvaDyva4TUK6pn/RH+jZFZ6/2RXk3rvSr3+IHwHrJ0BnccXGI5r9mz5sYYbncDvss9YqxqJHEmHtumTXMW96S768zYO7Pesrzn1YvsjflmgHQWCMkZfXDhM9rfwwjSTHjpLOTW6xj7F427ldQCa7XUre8/UZ79/ZIt2hISkBC/dx82ZhtazSGg7hPGpOuGSqQVEGhgdUIHD/LxaHypRhM1vvw9upp+/Dz9cPWQD+nigk5xPk2m48lDPhxmG3pkF2jhgr12IUB6MeNA+3T1Y3WYp2xDjXBX3Q9aUMN2Rz7PpCDPVUE1YtRVZ8liIVSue8tbRPM/AAD//3SFvhEMBAAA" time=2026-03-26T09:53:27.751Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-26T09:53:27.751Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | e2e | failed | #4510 (merged) | 1 | e2e (1)VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:159]: Unexpected error:
<*errors.joinError | 0xc000b2a570>:
VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then
{
errs: [
<*fmt.wrapError | 0xc000f26be0>{
msg: "VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then",
err: <*errors.errorString | 0xc0007ff6e0>{
s: "the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then",
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:159]: Unexpected error:
<*errors.joinError | 0xc000b2a570>:
VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then
{
errs: [
<*fmt.wrapError | 0xc000f26be0>{
msg: "VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then",
err: <*errors.errorString | 0xc0007ff6e0>{
s: "the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then",
},
},
],
}
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4641 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4505 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4626 (closed) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4600 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | e2e | failed | #4473 (merged) | 1 | e2e (1)timeout during CreateHCPClusterAndWait; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
<*fmt.wrapErrors | 0xc001287800>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-cndfg8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-cndfg8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00140f540>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
<*fmt.wrapErrors | 0xc001287800>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-cndfg8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-cndfg8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc00140f540>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8",
},
<context.deadlineExceededError>{},
],
}
occurred |
|
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4640 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | n/a | passed | #4578 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | n/a | passed | #4631 (merged) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | unknown | failed | #4631 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | unknown | failed | #4631 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | unknown | failed | #4631 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | unknown | failed | #4631 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | unknown | failed | #4631 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | n/a | passed | #4638 (closed) | 0 | n/a | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | unknown | failed | #4631 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | unknown | failed | #4631 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | unknown | failed | #4631 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | unknown | failed | #4631 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
n/a | unknown | failed | #4473 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | unknown | failed | #4631 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | unknown | failed | #4626 (closed) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel batch |
n/a | unknown | failed | #4626 (closed) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
n/a | unknown | failed | #4605 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| n/a | pull-ci-Azure-ARO-HCP-main-e2e-parallel batch |
n/a | provision | failed | #4605 (merged) | 1 | provision (1)ERROR CODE: DeploymentFailed; detail code AnmPreconditionFailedException; detail message Error while processing network security perimeter association.; provider Microsoft.Network Show raw failuretime=2026-03-26T01:04:22.906Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-26T01:04:24.455Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-26T01:04:25.942Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=b222a08f24481e15cc453d548a8685e1bf6587357f329bda2c50ffce3e76ca14
time=2026-03-26T01:16:28.345Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j7183104-svc/providers/Microsoft.Resources/deployments/b222a08f24481e15cc453d548a8685e1bf6587357f329bda2c50ffce3e76ca14/operationStatuses/08584271198203946776\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7183104-svc/providers/Microsoft.Resources/deployments/profile-imku7h4w54arm\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7183104-svc/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j7183104-svc/resourceAssociations/nsp-j7183104-svc-ra-b7209b60-e826-5723-b8f6-038fa1dd39d5\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AnmPreconditionFailedException\\\",\\r\\n \\\"message\\\": \\\"Error while processing request for association /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7183104-svc/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j7183104-svc/resourceAssociations/nsp-j7183104-svc-ra-b7209b60-e826-5723-b8f6-038fa1dd39d5, Error message: There is already an operation in progress which requires exclusive lock on this service arohcpprow-rp-j7183104. Please retry the operation after sometime.\\\\r\\\\nActivityId: addf7b91-7865-44f6-bc6c-2062a8ac9d18, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" |