<
Single Day: Mar 26▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: DEV

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 57b0fed · merge ef0f1b1
n/a e2e failed ⚠#4652 (merged) 1
e2e (1)
VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4
22:33:53 UTC · failed at e2e · test Customer should perform a control plane y-stream upgrade from 4.20 minor to 4.21 minor · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:171]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc000114e88>: 
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0, targetMinor=4.21.0) failed: clusterversion status.history has no version in target minor "4.21.0"
    {
        errs: [
            <*fmt.wrapError | 0xc000a1a7a0>{
                msg: "VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0, targetMinor=4.21.0) failed: clusterversion status.history has no version in target minor \"4.21.0\"",
                err: <*errors.errorString | 0xc0007aa580>{
                    s: "clusterversion status.history has no version in target minor \"4.21.0\"",
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:171]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc000114e88>: 
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0, targetMinor=4.21.0) failed: clusterversion status.history has no version in target minor "4.21.0"
    {
        errs: [
            <*fmt.wrapError | 0xc000a1a7a0>{
                msg: "VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0, targetMinor=4.21.0) failed: clusterversion status.history has no version in target minor \"4.21.0\"",
                err: <*errors.errorString | 0xc0007aa580>{
                    s: "clusterversion status.history has no version in target minor \"4.21.0\"",
                },
            },
        ],
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head d3d1e55 · merge e23d7c6
n/a e2e failed #4640 (merged) 1
e2e (1)
read access verification failed: get pod <pod> kube-system: pods is forbidden: User "system:sre-break-glass:<uuid>" cannot list resource "pods" in API group "" in the namespace "<namespace>"
22:32:22 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:281]: Timed out after 30.001s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc001030540>: 
    read access verification failed: get pods -n kube-system: pods is forbidden: User "system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652" cannot list resource "pods" in API group "" in the namespace "kube-system"
    get configmaps -n kube-system: configmaps is forbidden: User "system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
    {
        msg: "read access verification failed: get pods -n kube-system: pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"\nget configmaps -n kube-system: configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
        err: <*errors.joinError | 0xc000fb1de8>{
            errs: [
                <*fmt.wrapError | 0xc001100b60>{
                    msg: "get pods -n kube-system: pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
                    err: <*errors.StatusError | 0xc001106b40>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                            },
                            Status: "Failure",
                            Message: "pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
                            Reason: "Forbidden",
                            Details: {Name: "", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
                            Code: 403,
                        },
                    },
                },
                <*fmt.wrapError | 0xc0010304e0>{
                    msg: "get configmaps -n kube-system: configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
                    err: <*errors.StatusError | 0xc001020780>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                            },
                            Status: "Failure",
                            Message: "configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
                            Reason: "Forbidden",
                            Details: {Name: "", Group: "", Kind: "configmaps", UID: "", Causes: nil, RetryAfterSeconds: 0},
                            Code: 403,
                        },
                    },
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:281]: Timed out after 30.001s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc001030540>: 
    read access verification failed: get pods -n kube-system: pods is forbidden: User "system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652" cannot list resource "pods" in API group "" in the namespace "kube-system"
    get configmaps -n kube-system: configmaps is forbidden: User "system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
    {
        msg: "read access verification failed: get pods -n kube-system: pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"\nget configmaps -n kube-system: configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
        err: <*errors.joinError | 0xc000fb1de8>{
            errs: [
                <*fmt.wrapError | 0xc001100b60>{
                    msg: "get pods -n kube-system: pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
                    err: <*errors.StatusError | 0xc001106b40>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                            },
                            Status: "Failure",
                            Message: "pods is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
                            Reason: "Forbidden",
                            Details: {Name: "", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
                            Code: 403,
                        },
                    },
                },
                <*fmt.wrapError | 0xc0010304e0>{
                    msg: "get configmaps -n kube-system: configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
                    err: <*errors.StatusError | 0xc001020780>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                            },
                            Status: "Failure",
                            Message: "configmaps is forbidden: User \"system:sre-break-glass:c209f8df-52ae-48fb-98ea-380f58b04652\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"",
                            Reason: "Forbidden",
                            Details: {Name: "", Group: "", Kind: "configmaps", UID: "", Causes: nil, RetryAfterSeconds: 0},
                            Code: 403,
                        },
                    },
                },
            ],
        },
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head db17c8e · merge db17c8e
n/a e2e failed ★#4607 (merged) 2
e2e (2)
ERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.; provider Microsoft.Network
19:00:53 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
    <*fmt.wrapError | 0xc000757760>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-subnet-reuse-cjvhd4" in resourcegroup="rg-cluster-nsg-subnet-reuse-nsb4p6" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-cjvhd4/operationStatuses/08584270527184429378
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "BadRequest",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n    \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n    \"details\": []\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-subnet-reuse-cjvhd4\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-nsb4p6\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-cjvhd4/operationStatuses/08584270527184429378\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000757740>{
            msg: "failed waiting for deployment \"customer-infra-cluster-subnet-reuse-cjvhd4\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-nsb4p6\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-cjvhd4/operationStatuses/08584270527184429378\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0008e8cc0>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
     ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
    <*fmt.wrapError | 0xc000757760>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-subnet-reuse-cjvhd4" in resourcegroup="rg-cluster-nsg-subnet-reuse-nsb4p6" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-cjvhd4/operationStatuses/08584270527184429378
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "BadRequest",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n    \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n    \"details\": []\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-subnet-reuse-cjvhd4\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-nsb4p6\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-cjvhd4/operationStatuses/08584270527184429378\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000757740>{
            msg: "failed waiting for deployment \"customer-infra-cluster-subnet-reuse-cjvhd4\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-nsb4p6\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-cjvhd4/operationStatuses/08584270527184429378\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-nsb4p6/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0008e8cc0>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
     ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.; provider Microsoft.Network
19:00:53 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
    <*fmt.wrapError | 0xc0014d4940>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-2x54nn" in resourcegroup="customer-rg-h9bxs7" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-h9bxs7/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-2x54nn/operationStatuses/08584270525680151780
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "BadRequest",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n    \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n    \"details\": []\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-2x54nn\" in resourcegroup=\"customer-rg-h9bxs7\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-h9bxs7/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-2x54nn/operationStatuses/08584270525680151780\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0014d4920>{
            msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-2x54nn\" in resourcegroup=\"customer-rg-h9bxs7\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-h9bxs7/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-2x54nn/operationStatuses/08584270525680151780\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000d84cf0>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
    <*fmt.wrapError | 0xc0014d4940>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-2x54nn" in resourcegroup="customer-rg-h9bxs7" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-h9bxs7/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-2x54nn/operationStatuses/08584270525680151780
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "BadRequest",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n    \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n    \"details\": []\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-2x54nn\" in resourcegroup=\"customer-rg-h9bxs7\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-h9bxs7/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-2x54nn/operationStatuses/08584270525680151780\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0014d4920>{
            msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-2x54nn\" in resourcegroup=\"customer-rg-h9bxs7\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-h9bxs7/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-2x54nn/operationStatuses/08584270525680151780\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h9bxs7/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000d84cf0>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 949f619 · merge d00be7a
n/a provision failed #4625 (merged) 1
provision (1)
Internal error occurred: failed calling webhook "validation.istio.io": failed to call webhook: Post "<url>": proxy error from localhost:9443 while dialing <ip>:<port>, code 500: 500 Internal Server Error
18:44:56 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SessionGate/service/deploy · suite step graph
Show raw failure
time=2026-03-26T19:27:42.856Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-26T19:27:43.551Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:571b690d-4bcd-4d64-8d4e-a350c2693d67 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j3442048] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j3442048.hcpsvc.osadev.cloud] image:map[digest:sha256:7b5ae3d95bc764234f0b78cbc111ee7528b6efb8b2a9027e6e6a707fb66cb5b9 registry:arohcpsvcdev.azurecr.io repository:ci-op-br33ciqz/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-br33ciqz/pipeline:pull workloadIdentityClientId:442e39d9-a6b5-4cab-9f5d-38d223e0b76e workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:a3867303-90c6-46e9-b871-b2d7f79da93a workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]"
time=2026-03-26T19:27:43.551Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-26T19:27:43.819Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy namespace=sessiongate
time=2026-03-26T19:27:44.123Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy dryRun=false
time=2026-03-26T19:27:44.123Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-26T19:27:44.236Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-26T19:27:51.065Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.6:15017, code 500: 500 Internal Server Error"
time=2026-03-26T19:27:51.065Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-26T19:27:51.065Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-26T19:27:52.732Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy release=sessiongate namespace=sessiongate status=failed description="Release \"sessiongate\" failed: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.6:15017, code 500: 500 Internal Server Error" values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:571b690d-4bcd-4d64-8d4e-a350c2693d67 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j3442048] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j3442048.hcpsvc.osadev.cloud] image:map[digest:sha256:7b5ae3d95bc764234f0b78cbc111ee7528b6efb8b2a9027e6e6a707fb66cb5b9 registry:arohcpsvcdev.azurecr.io repository:ci-op-br33ciqz/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-br33ciqz/pipeline:pull workloadIdentityClientId:442e39d9-a6b5-4cab-9f5d-38d223e0b76e workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:a3867303-90c6-46e9-b871-b2d7f79da93a workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]"
time=2026-03-26T19:27:52.735Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5SUQU/cPhDF7/kUVk6stI5g//ypSsWBQqVKlaoVcKLqwXEeicHxpOMxCMSHr0LYEoVlF27OZOY3z+OnyTxEMSIltojqSFVGjJjSY4fKa1j54UJ1GIVdqOdqCP00Lf6FcIvwFImdsavw7FemlFL5OfjWWRxbSylIPld5RIyOQm0Erz6HmhOfooDPyE8z9PNZtyaYGi0G5KpwXYWlIEzeg9/otrlIe5gKrOFhxVHYAPnqQuVCvYFVrs34GGOiZwvzFJ2n+9Wcto/+zXxtweKunO3PjCtGbIaRmpuoXRRH2oWaEeOKdWx5mbwfXalL3m8RfA7LkCXTravAJ970vIkHbLex70mKQu3Zs6NPceWCmzxdLEbAwjDpxnaFeUiMwlI79tSzgd83vuMkDbF7MH2/JXln75+kek93U7EfBwSS5u3Jfb+4WJ5RWqNtvdvwJyFK3xBB+nd97W7NQ5I2SZopJ/v9JbtJJThAEL/1SyBmj+quAUOJaxHFtJ0qIXdAUDuVEfThnXyxuzjQu//pxcHF3ufDxafD/f3LfKaKQm3I+X/vMp/Nskd1TS6oGxeqIxcCeLS6KKiXjTVeVdMdlT2qjqn/+yJ0rjqqhuT3QebKUz1XLWI0dY8krsCqvB9dvkK0fwMAAP//DSzs0mAFAAA="
time=2026-03-26T19:27:52.737Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-26T19:27:52.737Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.6:15017, code 500: 500 Internal Server Error"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 3963f2a · merge 3963f2a
n/a n/a passed #4653 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head efa0d84 · merge ef0f1b1
n/a e2e failed ★#4652 (merged) 3
e2e (3)
ERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.; provider Microsoft.Network
18:11:11 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
    <*fmt.wrapError | 0xc000eae9e0>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-subnet-reuse-n58pg4" in resourcegroup="rg-cluster-nsg-subnet-reuse-wcrvrb" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-n58pg4/operationStatuses/08584270553481553506
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "BadRequest",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n    \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n    \"details\": []\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-subnet-reuse-n58pg4\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-wcrvrb\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-n58pg4/operationStatuses/08584270553481553506\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000eae9c0>{
            msg: "failed waiting for deployment \"customer-infra-cluster-subnet-reuse-n58pg4\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-wcrvrb\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-n58pg4/operationStatuses/08584270553481553506\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0009d2210>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
     ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
    <*fmt.wrapError | 0xc000eae9e0>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-subnet-reuse-n58pg4" in resourcegroup="rg-cluster-nsg-subnet-reuse-wcrvrb" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-n58pg4/operationStatuses/08584270553481553506
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "BadRequest",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n    \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n    \"details\": []\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-subnet-reuse-n58pg4\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-wcrvrb\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-n58pg4/operationStatuses/08584270553481553506\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000eae9c0>{
            msg: "failed waiting for deployment \"customer-infra-cluster-subnet-reuse-n58pg4\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-wcrvrb\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-n58pg4/operationStatuses/08584270553481553506\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-wcrvrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0009d2210>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
     ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.; provider Microsoft.Network
18:11:11 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
    <*fmt.wrapError | 0xc000046020>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-qdjsd4" in resourcegroup="customer-rg-nqljpg" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-nqljpg/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-qdjsd4/operationStatuses/08584270551842672141
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "BadRequest",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n    \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n    \"details\": []\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-qdjsd4\" in resourcegroup=\"customer-rg-nqljpg\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-nqljpg/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-qdjsd4/operationStatuses/08584270551842672141\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000046000>{
            msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-qdjsd4\" in resourcegroup=\"customer-rg-nqljpg\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-nqljpg/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-qdjsd4/operationStatuses/08584270551842672141\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0010ba450>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
    <*fmt.wrapError | 0xc000046020>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-qdjsd4" in resourcegroup="customer-rg-nqljpg" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-nqljpg/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-qdjsd4/operationStatuses/08584270551842672141
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "BadRequest",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n    \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n    \"details\": []\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-qdjsd4\" in resourcegroup=\"customer-rg-nqljpg\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-nqljpg/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-qdjsd4/operationStatuses/08584270551842672141\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000046000>{
            msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-qdjsd4\" in resourcegroup=\"customer-rg-nqljpg\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-nqljpg/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-qdjsd4/operationStatuses/08584270551842672141\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-nqljpg/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0010ba450>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift
18:11:11 UTC · failed at e2e · test Customer should perform a control plane y-stream upgrade from 4.20 minor to 4.21 minor · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:164]: Unexpected error:
    <*exported.ResponseError | 0xc0008f9ad0>: 
    PATCH https://rp.j3624320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-5pq6sw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-d4srhh
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InvalidRequestContent",
        StatusCode: 400,
        RawResponse: {
            Status: "400 Bad Request",
            StatusCode: 400,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "X-Ms-Request-Id": [
                    "7a22645f-db58-4a17-b26f-09787f0217a0",
                ],
                "Date": [
                    "Thu, 26 Mar 2026 19:36:23 GMT",
                ],
                "Content-Length": ["188"],
                "X-Envoy-Upstream-Service-Time": ["65"],
                "Server": ["istio-envoy"],
                "Content-Type": ["application/json"],
                "X-Ms-Error-Code": [
                    "InvalidRequestContent",
                ],
            },
            Body: <*exported.nopClosingBytesReader | 0xc000ca4ec0>{
                s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 82, 101, 113, 117, 105, 114, 101, 100, 32, 118, 97, 108, 117, 101, 58, 32, 102, 105, 101, 108, 100, 32, 99, 97, 110, 110, 111, 116, 32, 98, 101, 32, 110, 117, 108, 108, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 46, 112, 108, 97, 116, 102, 111, 114, 109, 46, 118, 110, 101, 116, 73, 110, 116, 101, 103, 114, 97, 116, 105, 111, 110, 83, 117, 98, 110, 101, 116, 73, 100, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                i: 0,
            },
            ContentLength: 188,
            TransferEncoding: nil,
            Close: false,
            Uncompressed: false,
            Trailer: nil,
            Request: {
                Method: "PATCH",
                URL: {
                    Scheme: "https",
                    Opaque: "",
                    User: nil,
                    Host: "rp.j3624320.hcpsvc.osadev.cloud",
                    Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-5pq6sw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-d4srhh",
                    RawPath: "",
                    OmitHost: false,
                    ForceQuery: false,
                    RawQuery: "api-version=2025-12-23-preview",
                    Fragment: "",
                    RawFragment: "",
                },
                Proto: "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header: {
                    "Authorization": ["redacted"],
                    "X-Ms-Identity-Url": [
                        "https://dummyhost.identity.azure.net",
                    ],
                    "Accept": ["application/json"],
                    "Content-Length": ["66"],
                    "Content-Type": ["application/json"],
                    "User-Agent": [
                        "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
                    ],
                    "X-Ms-Arm-Resource-System-Data": [
                        "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-26T19:36:23Z\"}",
                    ],
                },
                Body: <*http.readTrackingBody | 0xc000c8a7f8>{
                    ReadCloser: <*runtime.retryableRequestBody | 0xc0007e9380>{
                        body: <exported.nopCloser>{
                            ReadSeeker: <*bytes.Reader | 0xc0004f7050>{
                                s: "{\"properties\":{\"version\":{\"channelGroup\":\"stable\",\"id\":\"4.21.6\"}}}",
                                i: 66,
                                prevRune: -1,
      ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:164]: Unexpected error:
    <*exported.ResponseError | 0xc0008f9ad0>: 
    PATCH https://rp.j3624320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-5pq6sw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-d4srhh
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InvalidRequestContent",
        StatusCode: 400,
        RawResponse: {
            Status: "400 Bad Request",
            StatusCode: 400,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "X-Ms-Request-Id": [
                    "7a22645f-db58-4a17-b26f-09787f0217a0",
                ],
                "Date": [
                    "Thu, 26 Mar 2026 19:36:23 GMT",
                ],
                "Content-Length": ["188"],
                "X-Envoy-Upstream-Service-Time": ["65"],
                "Server": ["istio-envoy"],
                "Content-Type": ["application/json"],
                "X-Ms-Error-Code": [
                    "InvalidRequestContent",
                ],
            },
            Body: <*exported.nopClosingBytesReader | 0xc000ca4ec0>{
                s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 82, 101, 113, 117, 105, 114, 101, 100, 32, 118, 97, 108, 117, 101, 58, 32, 102, 105, 101, 108, 100, 32, 99, 97, 110, 110, 111, 116, 32, 98, 101, 32, 110, 117, 108, 108, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 46, 112, 108, 97, 116, 102, 111, 114, 109, 46, 118, 110, 101, 116, 73, 110, 116, 101, 103, 114, 97, 116, 105, 111, 110, 83, 117, 98, 110, 101, 116, 73, 100, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                i: 0,
            },
            ContentLength: 188,
            TransferEncoding: nil,
            Close: false,
            Uncompressed: false,
            Trailer: nil,
            Request: {
                Method: "PATCH",
                URL: {
                    Scheme: "https",
                    Opaque: "",
                    User: nil,
                    Host: "rp.j3624320.hcpsvc.osadev.cloud",
                    Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-5pq6sw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-d4srhh",
                    RawPath: "",
                    OmitHost: false,
                    ForceQuery: false,
                    RawQuery: "api-version=2025-12-23-preview",
                    Fragment: "",
                    RawFragment: "",
                },
                Proto: "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header: {
                    "Authorization": ["redacted"],
                    "X-Ms-Identity-Url": [
                        "https://dummyhost.identity.azure.net",
                    ],
                    "Accept": ["application/json"],
                    "Content-Length": ["66"],
                    "Content-Type": ["application/json"],
                    "User-Agent": [
                        "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
                    ],
                    "X-Ms-Arm-Resource-System-Data": [
                        "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-26T19:36:23Z\"}",
                    ],
                },
                Body: <*http.readTrackingBody | 0xc000c8a7f8>{
                    ReadCloser: <*runtime.retryableRequestBody | 0xc0007e9380>{
                        body: <exported.nopCloser>{
                            ReadSeeker: <*bytes.Reader | 0xc0004f7050>{
                                s: "{\"properties\":{\"version\":{\"channelGroup\":\"stable\",\"id\":\"4.21.6\"}}}",
                                i: 66,
                                prevRune: -1,
      ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head c90b1b8
n/a provision failed #4638 (closed) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid
17:46:37 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-26T18:16:27.487Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-26T18:16:30.165Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-26T18:16:31.296Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=6370d42ed2cab467b9d8a9e641c179817ed19b04da53274e67464cff44bd32f4
time=2026-03-26T18:18:01.839Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j4555904/providers/Microsoft.Resources/deployments/6370d42ed2cab467b9d8a9e641c179817ed19b04da53274e67464cff44bd32f4/operationStatuses/08584270578948601026\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4555904/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4555904/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j4555904\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 3e5d19b8-bdfb-4823-b567-c6d7077f42fb:3/26/2026 6:17:48 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head ebcd306 · merge 9feee09
n/a unknown failed #4004 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head c8fe032 · merge 0e19405
n/a unknown failed #4507 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 2a7d72c · merge 35d72d5
n/a unknown failed #4578 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head c8fe032 · merge 0e19405
n/a unknown failed #4507 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head f62a3d0 · merge db17c8e
n/a n/a passed #4607 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 84ba1cf · merge 0e19405
n/a unknown failed #4507 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head a9338f4 · merge 7e07b7c
n/a n/a passed #4630 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 9ec597a
n/a n/a passed #4638 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 6d02592 · merge 6d02592
n/a n/a passed #4651 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 24f6bb8 · merge 0e19405
n/a unknown failed #4507 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 05c202f · merge c66c503
n/a n/a passed #4645 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head a512416 · merge 0e19405
n/a unknown failed #4507 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 149d110 · merge e23d7c6
n/a e2e failed ⚠#4640 (merged) 1
e2e (1)
node pool upgrade verification failed: <node> (version 4
14:32:14 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.003s.
Expected success, but got an error:
    <*errors.errorString | 0xc00090ba80>: 
    node pool upgrade verification failed: q0f3h5j5u6h0w3l-npupgrade-4-20-kdc5m-k9xcn (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"q0f3h5j5u6h0w3l-npupgrade-4-20-bgvp5-gwb6f","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"q0f3h5j5u6h0w3l-npupgrade-4-20-bgvp5-zf4gf","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"q0f3h5j5u6h0w3l-npupgrade-4-20-kdc5m-k9xcn","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a9f74fb0da7fb9f7d70c01953cb70fdda6a22bb9182bc5358f5fd6d957f4f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef42cb417552fe1e38eedc30e92f55c8c50aae1aa1b0b7094ab56118efe0d96c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5dc767cd6267e006720af8cef541e23bc64d94ef9b1bf3a8b44b1fda5d9d8871","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dee9b6866db30456c277ff1f7a114dd4ce25bff8aced70218623af9423ec3cf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3c906f4908c692697ae58891ef463ee62a23d4356bf00786947cc1dc02e4d53","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78c23f5bccb6e82ade8428c8886ddc0291faec496590b2e6b5375ac6c0476dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ff94628410a702de0cb3e1a715917829b36c3def524dcc0850ac34a811f8f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:876fb57ffb255ed75a452405bcd1d116788a34b19264886e6e087ff544c17083","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ff360b712000d39cbb58158186ff0a18764278e6fa98ef9f21d113b978d9a79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77d8ddf2957025a39616328b5cf2b012de6cedce63fb79eebc5270d7b3401574","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec94607aa5fce47591b1ed1939e86a0e1c43cc8ff439f42662c9ed32e414595e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2962c6c31d7078b34cf7ba0c9b0ea685dc539cea67b06bd38b6ac566298bb790","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:321163a14934467a86e358bbb3115fbe104ad814bcaf7423f259eb82150da703","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7376a666810d810549b4e270d9a2fd6bcfd8799370ff52476a4f799cd4e287d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f426776d3ef70b585018b8d813f5eedd70373536681c6240df224ebf9ff6329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2906e0869191da12130ed79a57221aae735ef026d0f909534c7cbfbca131128","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3d68a9523f4c5c22a48dfa151bbfde78b2f360697afe3b9d8bfb53e8c359bf95","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7983b528aa73931fdaa817858ecd416dea0ec5a6917bebf5bd11b4eb83c752c9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f13a1e673c882ee926e97270249a20d3c40797a8c7c5abb2fda9c98c24a2eb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d4067116d68c1aabb5ee21c4addce05f6849f104d7a7f87f5f1fcef7990a3aee","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2af652f68b17ed49c2565928e4c101880e949c4484c6ee958793a8e668e47bea","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e199029f436d7247195d0c5b22909cb874ac67bd87d84c597a83ef7bd0d87ab0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:41e0dcd2f91df9234249f168b3fbe6d55cfdb3ebd2e598d5edfa47f905c046cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9998ff90e081aac492498314bd02225cc3a8636d8c714ce8b618f4ee1de28c21","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a054cfe1b8330bd57d762921ca5cd281ad2fdd7994fd3093ffadb5d767a564","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b369532d212af90d1232bcf0a84bc2616106733f4eae8d4aaf69fad595e1d67"]}]
    {
        s: "node pool upgrade verification failed: q0f3h5j5u6h0w3l-npupgrade-4-20-kdc5m-k9xcn (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"q0f3h5j5u6h0w3l-npupgrade-4-20-bgvp5-gwb6f\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9\",\"arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda\",\"arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.003s.
Expected success, but got an error:
    <*errors.errorString | 0xc00090ba80>: 
    node pool upgrade verification failed: q0f3h5j5u6h0w3l-npupgrade-4-20-kdc5m-k9xcn (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"q0f3h5j5u6h0w3l-npupgrade-4-20-bgvp5-gwb6f","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"q0f3h5j5u6h0w3l-npupgrade-4-20-bgvp5-zf4gf","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"q0f3h5j5u6h0w3l-npupgrade-4-20-kdc5m-k9xcn","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a9f74fb0da7fb9f7d70c01953cb70fdda6a22bb9182bc5358f5fd6d957f4f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef42cb417552fe1e38eedc30e92f55c8c50aae1aa1b0b7094ab56118efe0d96c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5dc767cd6267e006720af8cef541e23bc64d94ef9b1bf3a8b44b1fda5d9d8871","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dee9b6866db30456c277ff1f7a114dd4ce25bff8aced70218623af9423ec3cf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3c906f4908c692697ae58891ef463ee62a23d4356bf00786947cc1dc02e4d53","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78c23f5bccb6e82ade8428c8886ddc0291faec496590b2e6b5375ac6c0476dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ff94628410a702de0cb3e1a715917829b36c3def524dcc0850ac34a811f8f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:876fb57ffb255ed75a452405bcd1d116788a34b19264886e6e087ff544c17083","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ff360b712000d39cbb58158186ff0a18764278e6fa98ef9f21d113b978d9a79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77d8ddf2957025a39616328b5cf2b012de6cedce63fb79eebc5270d7b3401574","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec94607aa5fce47591b1ed1939e86a0e1c43cc8ff439f42662c9ed32e414595e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2962c6c31d7078b34cf7ba0c9b0ea685dc539cea67b06bd38b6ac566298bb790","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:321163a14934467a86e358bbb3115fbe104ad814bcaf7423f259eb82150da703","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7376a666810d810549b4e270d9a2fd6bcfd8799370ff52476a4f799cd4e287d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f426776d3ef70b585018b8d813f5eedd70373536681c6240df224ebf9ff6329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2906e0869191da12130ed79a57221aae735ef026d0f909534c7cbfbca131128","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3d68a9523f4c5c22a48dfa151bbfde78b2f360697afe3b9d8bfb53e8c359bf95","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7983b528aa73931fdaa817858ecd416dea0ec5a6917bebf5bd11b4eb83c752c9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f13a1e673c882ee926e97270249a20d3c40797a8c7c5abb2fda9c98c24a2eb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d4067116d68c1aabb5ee21c4addce05f6849f104d7a7f87f5f1fcef7990a3aee","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2af652f68b17ed49c2565928e4c101880e949c4484c6ee958793a8e668e47bea","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e199029f436d7247195d0c5b22909cb874ac67bd87d84c597a83ef7bd0d87ab0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:41e0dcd2f91df9234249f168b3fbe6d55cfdb3ebd2e598d5edfa47f905c046cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9998ff90e081aac492498314bd02225cc3a8636d8c714ce8b618f4ee1de28c21","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a054cfe1b8330bd57d762921ca5cd281ad2fdd7994fd3093ffadb5d767a564","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b369532d212af90d1232bcf0a84bc2616106733f4eae8d4aaf69fad595e1d67"]}]
    {
        s: "node pool upgrade verification failed: q0f3h5j5u6h0w3l-npupgrade-4-20-kdc5m-k9xcn (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"q0f3h5j5u6h0w3l-npupgrade-4-20-bgvp5-gwb6f\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9\",\"arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda\",\"arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head bee1a1c · merge c66c503
n/a unknown failed #4645 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head a3c49a0 · merge a3c49a0
n/a n/a passed #4510 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 56bf314 · merge 56bf314
n/a n/a passed #4485 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 56bf314 · merge 56bf314
n/a e2e failed #4485 (merged) 1
e2e (1)
timeout during CreateHCPClusterAndWait; context deadline exceeded
13:00:06 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc001444f90>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-2nbkt4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-2nbkt4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009456e0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4",
            },
            <*fmt.wrapErrors | 0xc001444f60>{
                msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-2nbkt4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0009456e0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc001444f90>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-2nbkt4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-2nbkt4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009456e0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4",
            },
            <*fmt.wrapErrors | 0xc001444f60>{
                msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-2nbkt4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0009456e0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-2nbkt4",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 56bf314 · merge 56bf314
n/a n/a passed #4485 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 7c2f4e3 · merge 7c2f4e3
n/a n/a passed #4648 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 9f3a000 · merge c168b54
n/a n/a passed #4577 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head cc6cdef · merge cc6cdef
n/a n/a passed #4647 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 88bf2d4 · merge 8b7576c
n/a e2e failed #4473 (merged) 1
e2e (1)
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:54:43 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011d91a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-48kgkn" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-48kgkn\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001196aa0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011d91a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-48kgkn" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-48kgkn\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001196aa0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-48kgkn",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head a3c49a0 · merge a3c49a0
n/a n/a passed #4510 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head ec8c17a · merge ec8c17a
n/a n/a passed #4505 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head b13100f
n/a n/a passed #4615 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 92e0c79 · merge c66c503
n/a n/a passed #4645 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 56bf314 · merge 56bf314
n/a n/a passed #4485 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 953df9e · merge 953df9e
n/a n/a passed #4605 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head ec8c17a · merge ec8c17a
n/a provision failed #4505 (merged) 1
provision (1)
Internal error occurred: failed calling webhook "validation.istio.io": failed to call webhook: Post "<url>": EOF
10:07:58 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SessionGate/service/deploy · suite step graph
Show raw failure
time=2026-03-26T10:48:23.528Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-26T10:48:24.177Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:2d906c8b-39a3-4047-8d40-cb975605ba4a cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j8294784] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j8294784.hcpsvc.osadev.cloud] image:map[digest:sha256:50b5fb5bf5e6af13c266dff43d6597d4e81da69b3fcc69a8884876fab568ea81 registry:arohcpsvcdev.azurecr.io repository:ci-op-6wcrgjtb/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-6wcrgjtb/pipeline:pull workloadIdentityClientId:277dd18f-32d4-4cb3-9d0f-7a85c95829b6 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:3301eb93-f945-4afd-b88e-96b8cc9739ba workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]"
time=2026-03-26T10:48:24.177Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-26T10:48:24.382Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy namespace=sessiongate
time=2026-03-26T10:48:24.693Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy dryRun=false
time=2026-03-26T10:48:24.694Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-26T10:48:24.820Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-26T10:48:32.557Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": EOF"
time=2026-03-26T10:48:32.557Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-26T10:48:32.557Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-26T10:48:34.078Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy release=sessiongate namespace=sessiongate status=failed description="Release \"sessiongate\" failed: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": EOF" values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:2d906c8b-39a3-4047-8d40-cb975605ba4a cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j8294784] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j8294784.hcpsvc.osadev.cloud] image:map[digest:sha256:50b5fb5bf5e6af13c266dff43d6597d4e81da69b3fcc69a8884876fab568ea81 registry:arohcpsvcdev.azurecr.io repository:ci-op-6wcrgjtb/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-6wcrgjtb/pipeline:pull workloadIdentityClientId:277dd18f-32d4-4cb3-9d0f-7a85c95829b6 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:3301eb93-f945-4afd-b88e-96b8cc9739ba workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]"
time=2026-03-26T10:48:34.081Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6RUQWvbTBS861csOsXgFf6cED5cckiTQqFQjOtTSg+r1VjeeLVPvLeycfCPL4rtRji249Lb6mlmtPM0TOIRFUOoYQtRd6ow0USTe1xR/gwbv7lQjCSyC2VfbUffTYU/IywRXidSG7sf934mSimV/gAvncW9tdSEmPZVKhBxFEoT8e5xy3nwjUTwhPwhQu/OujLBlKiwldwTW8ZnFwoXykOipRCZvAfr/Cji7zQ8TAHW8LCxvc95zd0SLnP/iNrTem/tH/DagqObOdueGTOGzMEtyCxEO4mOtAslQ2Svdd/EObF7Ma2pMXln169472l1yDh6mbMCFSI7K1o+NnVWJ1Ccn1721+l0PKHmiPTxsDUSqZrs8v+ImQuu/WgHLlmHlxkmPbd1Zl4aRmapOkzg6dicidzlWTu1McvjxvtOdOvG+w+DaRlxzLR0BfjBm3c/Voutj0cm+fUpWTQ5OCBCvrQlIMlGreZgqOgqSDRVrXLEFRDUVWEi2vFVOhwMb/XgWg9vp/8NRjf/j4Y3T2lPZZk6g7kePqW9XrJRz+SCWrhQ3LkQwJ3qoqDeGqtbVYcdlWxUzdS+fbtoX9VUbMGXifSVp7KvKoiYspUkLsAqX3fMFxD7OwAA//8kk60oYAUAAA=="
time=2026-03-26T10:48:34.082Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-26T10:48:34.083Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": EOF"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 1c28bed
n/a n/a passed #4644 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head ad17887 · merge e23d7c6
n/a n/a passed #4640 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 5e719eb · merge 5e719eb
n/a n/a passed #4643 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 9f3a000 · merge c168b54
n/a e2e failed #4577 (merged) 1
e2e (1)
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
09:24:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124]: Unexpected error:
    <*fmt.wrapErrors | 0xc000d048d0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b, error: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-vsdn4b" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b, error: failed waiting for nodepool=\"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vsdn4b\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0002c7a30>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b",
            },
            <*fmt.wrapError | 0xc00145e300>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vsdn4b\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124]: Unexpected error:
    <*fmt.wrapErrors | 0xc000d048d0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b, error: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-vsdn4b" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b, error: failed waiting for nodepool=\"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vsdn4b\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0002c7a30>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-vsdn4b",
            },
            <*fmt.wrapError | 0xc00145e300>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vsdn4b\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 5fe9081 · merge 5fe9081
n/a n/a passed #4642 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head a76de5c · merge 1d16d50
n/a provision failed #4446 (merged) 1
provision (1)
resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress context deadline exceeded
09:05:56 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RP.Backend/service/deploy · suite step graph
Show raw failure
time=2026-03-26T09:48:18.147Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-26T09:48:18.853Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[armPermissionsManagerIdentity:map[certName:armHelperCert2 clientId:3331e670-0804-48e8-a086-6241671ddc93 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj1912448.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj1912448.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clusterScopedIdentitiesConfig:map[roleSetName:dev] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:e9330836-104c-4ff6-8896-5e80a6561853 databaseName:arohcpprow-rp-j1912448 databaseUrl:https://arohcpprow-rp-j1912448.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-icg9x6nq/pipeline@sha256:d7affa8c94e1be1641298f474a1a1409323fb682a03fba66b2735fdfe48a905a replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-icg9x6nq/pipeline:pull workloadIdentityClientId:bbe90c53-94c0-41a0-a862-22f9c8836d86 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:e9330836-104c-4ff6-8896-5e80a6561853 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-26T09:48:18.853Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-26T09:48:18.912Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp
time=2026-03-26T09:48:19.026Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false
time=2026-03-26T09:48:19.026Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-26T09:48:19.137Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-26T09:53:23.504Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded"
time=2026-03-26T09:53:23.504Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-26T09:53:23.504Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-26T09:53:27.747Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy release=aro-hcp-backend-dev namespace=aro-hcp status=failed description="Release \"aro-hcp-backend-dev\" failed: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[armPermissionsManagerIdentity:map[certName:armHelperCert2 clientId:3331e670-0804-48e8-a086-6241671ddc93 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj1912448.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj1912448.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clusterScopedIdentitiesConfig:map[roleSetName:dev] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:e9330836-104c-4ff6-8896-5e80a6561853 databaseName:arohcpprow-rp-j1912448 databaseUrl:https://arohcpprow-rp-j1912448.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-icg9x6nq/pipeline@sha256:d7affa8c94e1be1641298f474a1a1409323fb682a03fba66b2735fdfe48a905a replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-icg9x6nq/pipeline:pull workloadIdentityClientId:bbe90c53-94c0-41a0-a862-22f9c8836d86 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:e9330836-104c-4ff6-8896-5e80a6561853 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-26T09:53:27.748Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/4xTTW8aMRS876942hNI64hCGzVUOaRpT1Uq1PSUqgevPV0cvH6rZy+IiB9fLSRhhSDhOm9m3oc9mUciQeRWDCJdk9VJJ116DLh8hEk/XLDTmMSFqqAd9FPXeIWwRNgisdHmBR7+yYiI8lsO/1x1p5u8oLzUZoFglX5qBUrakFwNZbaUrq6F1dw0efGG9iT7F3t0sIe2EAUPkxyHE8yvLlgXqr7z+8JvaDyva4TUK6pn/RH+jZFZ6/2RXk3rvSr3+IHwHrJ0BnccXGI5r9mz5sYYbncDvss9YqxqJHEmHtumTXMW96S768zYO7Pesrzn1YvsjflmgHQWCMkZfXDhM9rfwwjSTHjpLOTW6xj7F427ldQCa7XUre8/UZ79/ZIt2hISkBC/dx82ZhtazSGg7hPGpOuGSqQVEGhgdUIHD/LxaHypRhM1vvw9upp+/Dz9cPWQD+nigk5xPk2m48lDPhxmG3pkF2jhgr12IUB6MeNA+3T1Y3WYp2xDjXBX3Q9aUMN2Rz7PpCDPVUE1YtRVZ8liIVSue8tbRPM/AAD//3SFvhEMBAAA"
time=2026-03-26T09:53:27.751Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-03-26T09:53:27.751Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head a3c49a0 · merge a3c49a0
n/a e2e failed #4510 (merged) 1
e2e (1)
VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then
09:04:00 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:159]: Unexpected error:
    <*errors.joinError | 0xc000b2a570>: 
    VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then
    {
        errs: [
            <*fmt.wrapError | 0xc000f26be0>{
                msg: "VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then",
                err: <*errors.errorString | 0xc0007ff6e0>{
                    s: "the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:159]: Unexpected error:
    <*errors.joinError | 0xc000b2a570>: 
    VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then
    {
        errs: [
            <*fmt.wrapError | 0xc000f26be0>{
                msg: "VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then",
                err: <*errors.errorString | 0xc0007ff6e0>{
                    s: "the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then",
                },
            },
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 1192658 · merge 6a659af
n/a n/a passed #4641 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ec8c17a · merge ec8c17a
n/a n/a passed #4505 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 559c545
n/a n/a passed #4626 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 7de281f · merge 7de281f
n/a n/a passed #4600 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 8e38f2d · merge 8b7576c
n/a e2e failed #4473 (merged) 1
e2e (1)
timeout during CreateHCPClusterAndWait; context deadline exceeded
07:30:40 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
    <*fmt.wrapErrors | 0xc001287800>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-cndfg8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-cndfg8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00140f540>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:107]: Unexpected error:
    <*fmt.wrapErrors | 0xc001287800>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-cndfg8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-cndfg8\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00140f540>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cndfg8",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head ad17887 · merge e23d7c6
n/a n/a passed #4640 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head c34ea97 · merge 35d72d5
n/a n/a passed #4578 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c0f5f62 · merge c0f5f62
n/a n/a passed #4631 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c0f5f62 · merge c0f5f62
n/a unknown failed #4631 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c0f5f62 · merge c0f5f62
n/a unknown failed #4631 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c0f5f62 · merge c0f5f62
n/a unknown failed #4631 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c0f5f62 · merge c0f5f62
n/a unknown failed #4631 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c0f5f62 · merge c0f5f62
n/a unknown failed #4631 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 5ae5330
n/a n/a passed #4638 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c0f5f62 · merge c0f5f62
n/a unknown failed #4631 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c0f5f62 · merge c0f5f62
n/a unknown failed #4631 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c0f5f62 · merge c0f5f62
n/a unknown failed #4631 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c0f5f62 · merge c0f5f62
n/a unknown failed #4631 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 8e38f2d · merge 8b7576c
n/a unknown failed #4473 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c0f5f62 · merge c0f5f62
n/a unknown failed #4631 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 559c545
n/a unknown failed #4626 (closed) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 559c545
n/a unknown failed #4626 (closed) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 953df9e · merge 953df9e
n/a unknown failed #4605 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 953df9e · merge 953df9e
n/a provision failed #4605 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code AnmPreconditionFailedException; detail message Error while processing network security perimeter association.; provider Microsoft.Network
00:36:48 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph
Show raw failure
time=2026-03-26T01:04:22.906Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-26T01:04:24.455Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-26T01:04:25.942Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=b222a08f24481e15cc453d548a8685e1bf6587357f329bda2c50ffce3e76ca14
time=2026-03-26T01:16:28.345Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j7183104-svc/providers/Microsoft.Resources/deployments/b222a08f24481e15cc453d548a8685e1bf6587357f329bda2c50ffce3e76ca14/operationStatuses/08584271198203946776\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7183104-svc/providers/Microsoft.Resources/deployments/profile-imku7h4w54arm\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7183104-svc/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j7183104-svc/resourceAssociations/nsp-j7183104-svc-ra-b7209b60-e826-5723-b8f6-038fa1dd39d5\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"AnmPreconditionFailedException\\\",\\r\\n                \\\"message\\\": \\\"Error while processing request for association /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7183104-svc/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j7183104-svc/resourceAssociations/nsp-j7183104-svc-ra-b7209b60-e826-5723-b8f6-038fa1dd39d5, Error message: There is already an operation in progress which requires exclusive lock on this service arohcpprow-rp-j7183104. Please retry the operation after sometime.\\\\r\\\\nActivityId: addf7b91-7865-44f6-bc6c-2062a8ac9d18, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"