CIHealth Run Log

Environment: DEV

Time (UTC)	Runtime	Job	Region	Failed at	Result	PR	Failed tests	Details
23:51:04	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 5a03840 · merge b541ca7	n/a	e2e	failed	★#4196 (merged)	4	e2e (4) ERROR CODE: InternalServerError; detail code AuthorizationPermissionMismatch; provider Microsoft.RedHatOpenShift 23:51:04 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error: <fmt.wrapError \| 0xc000f1e780>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-jsnx7r" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/225ca079-6d7a-4934-af01-b741ca97f4ee -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/225ca079-6d7a-4934-af01-b741ca97f4ee", "name": "225ca079-6d7a-4934-af01-b741ca97f4ee", "status": "Failed", "startTime": "2026-02-26T00:54:19.286966114Z", "endTime": "2026-02-26T00:57:01.705617058Z", "error": { "code": "InternalServerError", "message": "GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:d028f085-801e-00ad-72ba-a6c4ef000000\nTime:2026-02-26T00:56:09.4065931Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster basic-cluster: failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-jsnx7r\" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/225ca079-6d7a-4934-af01-b741ca97f4ee\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/225ca079-6d7a-4934-af01-b741ca97f4ee\",\n \"name\": \"225ca079-6d7a-4934-af01-b741ca97f4ee\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-26T00:54:19.286966114Z\",\n \"endTime\": \"2026-02-26T00:57:01.705617058Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:d028f085-801e-00ad-72ba-a6c4ef000000\\nTime:2026-02-26T00:56:09.4065931Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f1e760>{ msg: "failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-jsnx7r\" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/225ca079-6d7a-4934-af01-b741ca97f4ee\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/225ca079-6d7a-4934-af01-b741ca97f4ee\",\n \"name\": \"225ca079-6d7a-4934-af01-b741ca97f4ee\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-26T00:54:19.286966114Z\",\n \"endTime\": \"2026-02-26T00:57:01.705617058Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:d028f085-801e-00ad-72ba-a6c4ef000000\\nTime:2026-02-26T00:56:09.4065931Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error: <fmt.wrapError \| 0xc000f1e780>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-jsnx7r" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/225ca079-6d7a-4934-af01-b741ca97f4ee -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/225ca079-6d7a-4934-af01-b741ca97f4ee", "name": "225ca079-6d7a-4934-af01-b741ca97f4ee", "status": "Failed", "startTime": "2026-02-26T00:54:19.286966114Z", "endTime": "2026-02-26T00:57:01.705617058Z", "error": { "code": "InternalServerError", "message": "GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:d028f085-801e-00ad-72ba-a6c4ef000000\nTime:2026-02-26T00:56:09.4065931Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster basic-cluster: failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-jsnx7r\" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/225ca079-6d7a-4934-af01-b741ca97f4ee\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/225ca079-6d7a-4934-af01-b741ca97f4ee\",\n \"name\": \"225ca079-6d7a-4934-af01-b741ca97f4ee\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-26T00:54:19.286966114Z\",\n \"endTime\": \"2026-02-26T00:57:01.705617058Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:d028f085-801e-00ad-72ba-a6c4ef000000\\nTime:2026-02-26T00:56:09.4065931Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f1e760>{ msg: "failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-jsnx7r\" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/225ca079-6d7a-4934-af01-b741ca97f4ee\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/225ca079-6d7a-4934-af01-b741ca97f4ee\",\n \"name\": \"225ca079-6d7a-4934-af01-b741ca97f4ee\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-26T00:54:19.286966114Z\",\n \"endTime\": \"2026-02-26T00:57:01.705617058Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:d028f085-801e-00ad-72ba-a6c4ef000000\\nTime:2026-02-26T00:56:09.4065931Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code AuthorizationPermissionMismatch; provider Microsoft.RedHatOpenShift 23:51:04 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error: <fmt.wrapError \| 0xc000d74d40>: failed to create HCP cluster ea-listhdnd74: failed waiting for cluster="ea-listhdnd74" in resourcegroup="ea-list-cm8692" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a5ca75-3490-45ee-b725-bdab9d1c2f87 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a5ca75-3490-45ee-b725-bdab9d1c2f87", "name": "19a5ca75-3490-45ee-b725-bdab9d1c2f87", "status": "Failed", "startTime": "2026-02-26T00:54:08.817035844Z", "endTime": "2026-02-26T00:57:01.558964391Z", "error": { "code": "InternalServerError", "message": "GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:d028ec26-801e-00ad-74ba-a6c4ef000000\nTime:2026-02-26T00:56:09.2186001Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-listhdnd74: failed waiting for cluster=\"ea-listhdnd74\" in resourcegroup=\"ea-list-cm8692\" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a5ca75-3490-45ee-b725-bdab9d1c2f87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a5ca75-3490-45ee-b725-bdab9d1c2f87\",\n \"name\": \"19a5ca75-3490-45ee-b725-bdab9d1c2f87\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-26T00:54:08.817035844Z\",\n \"endTime\": \"2026-02-26T00:57:01.558964391Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:d028ec26-801e-00ad-74ba-a6c4ef000000\\nTime:2026-02-26T00:56:09.2186001Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000d74d20>{ msg: "failed waiting for cluster=\"ea-listhdnd74\" in resourcegroup=\"ea-list-cm8692\" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a5ca75-3490-45ee-b725-bdab9d1c2f87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a5ca75-3490-45ee-b725-bdab9d1c2f87\",\n \"name\": \"19a5ca75-3490-45ee-b725-bdab9d1c2f87\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-26T00:54:08.817035844Z\",\n \"endTime\": \"2026-02-26T00:57:01.558964391Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:d028ec26-801e-00ad-74ba-a6c4ef000000\\nTime:2026-02-26T00:56:09.2186001Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0x... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error: <fmt.wrapError \| 0xc000d74d40>: failed to create HCP cluster ea-listhdnd74: failed waiting for cluster="ea-listhdnd74" in resourcegroup="ea-list-cm8692" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a5ca75-3490-45ee-b725-bdab9d1c2f87 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a5ca75-3490-45ee-b725-bdab9d1c2f87", "name": "19a5ca75-3490-45ee-b725-bdab9d1c2f87", "status": "Failed", "startTime": "2026-02-26T00:54:08.817035844Z", "endTime": "2026-02-26T00:57:01.558964391Z", "error": { "code": "InternalServerError", "message": "GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:d028ec26-801e-00ad-74ba-a6c4ef000000\nTime:2026-02-26T00:56:09.2186001Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-listhdnd74: failed waiting for cluster=\"ea-listhdnd74\" in resourcegroup=\"ea-list-cm8692\" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a5ca75-3490-45ee-b725-bdab9d1c2f87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a5ca75-3490-45ee-b725-bdab9d1c2f87\",\n \"name\": \"19a5ca75-3490-45ee-b725-bdab9d1c2f87\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-26T00:54:08.817035844Z\",\n \"endTime\": \"2026-02-26T00:57:01.558964391Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:d028ec26-801e-00ad-74ba-a6c4ef000000\\nTime:2026-02-26T00:56:09.2186001Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000d74d20>{ msg: "failed waiting for cluster=\"ea-listhdnd74\" in resourcegroup=\"ea-list-cm8692\" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a5ca75-3490-45ee-b725-bdab9d1c2f87\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19a5ca75-3490-45ee-b725-bdab9d1c2f87\",\n \"name\": \"19a5ca75-3490-45ee-b725-bdab9d1c2f87\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-26T00:54:08.817035844Z\",\n \"endTime\": \"2026-02-26T00:57:01.558964391Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:d028ec26-801e-00ad-74ba-a6c4ef000000\\nTime:2026-02-26T00:56:09.2186001Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0x... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code AuthorizationPermissionMismatch; provider Microsoft.RedHatOpenShift 23:51:04 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc000721080>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-fc4gzl" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2261cdb-1994-4cbf-afe2-37e4ffb6f562 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2261cdb-1994-4cbf-afe2-37e4ffb6f562", "name": "f2261cdb-1994-4cbf-afe2-37e4ffb6f562", "status": "Failed", "startTime": "2026-02-26T00:54:12.525607564Z", "endTime": "2026-02-26T00:57:01.857734749Z", "error": { "code": "InternalServerError", "message": "PUT https://arohcpprowoidcj7503360.blob.core.windows.net/$web/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/2omknd2gh99pacnb5gakhposl180tdg0/openid/v1/jwks\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:d028f0fb-801e-00ad-58ba-a6c4ef000000\nTime:2026-02-26T00:56:09.6363965Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-fc4gzl\" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2261cdb-1994-4cbf-afe2-37e4ffb6f562\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2261cdb-1994-4cbf-afe2-37e4ffb6f562\",\n \"name\": \"f2261cdb-1994-4cbf-afe2-37e4ffb6f562\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-26T00:54:12.525607564Z\",\n \"endTime\": \"2026-02-26T00:57:01.857734749Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"PUT https://arohcpprowoidcj7503360.blob.core.windows.net/$web/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/2omknd2gh99pacnb5gakhposl180tdg0/openid/v1/jwks\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:d028f0fb-801e-00ad-58ba-a6c4ef000000\\nTime:2026-02-26T00:56:09.6363965Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000721060>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-fc4gzl\" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2261cdb-1994-4cbf-afe2-37e4ffb6f562\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2261cdb-1994-4cbf-afe2-37e4ffb6f562\",\n \"name\": \"f2261cdb-1994-4cbf-afe2-37e4ffb6f562\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-26T00:54:12.525607564Z\",\n \"endTime\": \"2026-02-26T00:57:01.857734749Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"PUT https://arohcpprowoidcj7503360.blob.core.windows.net/$web/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/2omknd2gh99pacnb5gakhposl180tdg0/openid/v1/jwks\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:d028f0fb-801e-00ad-58ba-a6c4ef000000\\nTime:2026-02-26T00:56:09.6363965Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n-------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc000721080>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-fc4gzl" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2261cdb-1994-4cbf-afe2-37e4ffb6f562 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2261cdb-1994-4cbf-afe2-37e4ffb6f562", "name": "f2261cdb-1994-4cbf-afe2-37e4ffb6f562", "status": "Failed", "startTime": "2026-02-26T00:54:12.525607564Z", "endTime": "2026-02-26T00:57:01.857734749Z", "error": { "code": "InternalServerError", "message": "PUT https://arohcpprowoidcj7503360.blob.core.windows.net/$web/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/2omknd2gh99pacnb5gakhposl180tdg0/openid/v1/jwks\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:d028f0fb-801e-00ad-58ba-a6c4ef000000\nTime:2026-02-26T00:56:09.6363965Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-fc4gzl\" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2261cdb-1994-4cbf-afe2-37e4ffb6f562\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2261cdb-1994-4cbf-afe2-37e4ffb6f562\",\n \"name\": \"f2261cdb-1994-4cbf-afe2-37e4ffb6f562\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-26T00:54:12.525607564Z\",\n \"endTime\": \"2026-02-26T00:57:01.857734749Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"PUT https://arohcpprowoidcj7503360.blob.core.windows.net/$web/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/2omknd2gh99pacnb5gakhposl180tdg0/openid/v1/jwks\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:d028f0fb-801e-00ad-58ba-a6c4ef000000\\nTime:2026-02-26T00:56:09.6363965Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000721060>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-fc4gzl\" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2261cdb-1994-4cbf-afe2-37e4ffb6f562\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2261cdb-1994-4cbf-afe2-37e4ffb6f562\",\n \"name\": \"f2261cdb-1994-4cbf-afe2-37e4ffb6f562\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-26T00:54:12.525607564Z\",\n \"endTime\": \"2026-02-26T00:57:01.857734749Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"PUT https://arohcpprowoidcj7503360.blob.core.windows.net/$web/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/2omknd2gh99pacnb5gakhposl180tdg0/openid/v1/jwks\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:d028f0fb-801e-00ad-58ba-a6c4ef000000\\nTime:2026-02-26T00:56:09.6363965Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n-------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code AuthorizationPermissionMismatch; provider Microsoft.RedHatOpenShift 23:51:04 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error: <fmt.wrapError \| 0xc0009189e0>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-xz7m8f" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1869cf79-9ebe-43a3-9d4f-06ed3bc51b03 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1869cf79-9ebe-43a3-9d4f-06ed3bc51b03", "name": "1869cf79-9ebe-43a3-9d4f-06ed3bc51b03", "status": "Failed", "startTime": "2026-02-26T00:54:29.822288857Z", "endTime": "2026-02-26T00:57:02.314815127Z", "error": { "code": "InternalServerError", "message": "GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:d028f05f-801e-00ad-4eba-a6c4ef000000\nTime:2026-02-26T00:56:09.3125409Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-xz7m8f\" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1869cf79-9ebe-43a3-9d4f-06ed3bc51b03\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1869cf79-9ebe-43a3-9d4f-06ed3bc51b03\",\n \"name\": \"1869cf79-9ebe-43a3-9d4f-06ed3bc51b03\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-26T00:54:29.822288857Z\",\n \"endTime\": \"2026-02-26T00:57:02.314815127Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:d028f05f-801e-00ad-4eba-a6c4ef000000\\nTime:2026-02-26T00:56:09.3125409Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000918980>{ msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-xz7m8f\" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1869cf79-9ebe-43a3-9d4f-06ed3bc51b03\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1869cf79-9ebe-43a3-9d4f-06ed3bc51b03\",\n \"name\": \"1869cf79-9ebe-43a3-9d4f-06ed3bc51b03\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-26T00:54:29.822288857Z\",\n \"endTime\": \"2026-02-26T00:57:02.314815127Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:d028f05f-801e-00ad-4eba-a6c4ef000000\\nTime:2026-02-26T00:56:09.3125409Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n-----------------------------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error: <fmt.wrapError \| 0xc0009189e0>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-xz7m8f" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1869cf79-9ebe-43a3-9d4f-06ed3bc51b03 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1869cf79-9ebe-43a3-9d4f-06ed3bc51b03", "name": "1869cf79-9ebe-43a3-9d4f-06ed3bc51b03", "status": "Failed", "startTime": "2026-02-26T00:54:29.822288857Z", "endTime": "2026-02-26T00:57:02.314815127Z", "error": { "code": "InternalServerError", "message": "GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:d028f05f-801e-00ad-4eba-a6c4ef000000\nTime:2026-02-26T00:56:09.3125409Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-xz7m8f\" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1869cf79-9ebe-43a3-9d4f-06ed3bc51b03\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1869cf79-9ebe-43a3-9d4f-06ed3bc51b03\",\n \"name\": \"1869cf79-9ebe-43a3-9d4f-06ed3bc51b03\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-26T00:54:29.822288857Z\",\n \"endTime\": \"2026-02-26T00:57:02.314815127Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:d028f05f-801e-00ad-4eba-a6c4ef000000\\nTime:2026-02-26T00:56:09.3125409Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000918980>{ msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-xz7m8f\" to finish creating: GET https://rp.j7503360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1869cf79-9ebe-43a3-9d4f-06ed3bc51b03\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1869cf79-9ebe-43a3-9d4f-06ed3bc51b03\",\n \"name\": \"1869cf79-9ebe-43a3-9d4f-06ed3bc51b03\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-26T00:54:29.822288857Z\",\n \"endTime\": \"2026-02-26T00:57:02.314815127Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj7503360.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:d028f05f-801e-00ad-4eba-a6c4ef000000\\nTime:2026-02-26T00:56:09.3125409Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n-----------------------------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
23:50:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head a02b279 · merge a02b279	n/a	n/a	passed	#4207 (merged)	0	n/a
23:50:33	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head dcf56c7 · merge dcf56c7	n/a	unknown	failed	#3780 (merged)	1	non-artifact-backed failure (no junit artifacts)
23:50:27	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 66b441c · merge 66b441c	n/a	n/a	passed	#3956 (merged)	0	n/a
23:41:06	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 438fac2 · merge 7cbe20c	n/a	n/a	passed	#4218 (merged)	0	n/a
23:10:53	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 9ff64c7 · merge ad7003e	n/a	n/a	passed	#4188 (merged)	0	n/a
22:53:05	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 096bc31 · merge 096bc31	n/a	n/a	passed	#4213 (merged)	0	n/a
22:49:52	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 3ca804d · merge 3ca804d	n/a	e2e	failed	★#3679 (merged)	2	e2e (2) Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed 22:49:52 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:153]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:153]: Cluster provisioning failed ERROR CODE: InternalServerError; detail code AuthorizationPermissionMismatch; provider Microsoft.RedHatOpenShift 22:49:52 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapError \| 0xc000dd6360>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-bz5pm9" to finish creating: GET https://rp.j0659584.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9b308276-5a29-4bdc-9917-0a6f5eb47ab9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9b308276-5a29-4bdc-9917-0a6f5eb47ab9", "name": "9b308276-5a29-4bdc-9917-0a6f5eb47ab9", "status": "Failed", "startTime": "2026-02-25T23:38:05.329111798Z", "endTime": "2026-02-25T23:40:46.498452335Z", "error": { "code": "InternalServerError", "message": "GET https://arohcpprowoidcj0659584.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:5b666743-501e-0081-38b0-a62840000000\nTime:2026-02-25T23:40:12.7324517Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-bz5pm9\" to finish creating: GET https://rp.j0659584.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9b308276-5a29-4bdc-9917-0a6f5eb47ab9\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9b308276-5a29-4bdc-9917-0a6f5eb47ab9\",\n \"name\": \"9b308276-5a29-4bdc-9917-0a6f5eb47ab9\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-25T23:38:05.329111798Z\",\n \"endTime\": \"2026-02-25T23:40:46.498452335Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj0659584.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:5b666743-501e-0081-38b0-a62840000000\\nTime:2026-02-25T23:40:12.7324517Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000dd6340>{ msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-bz5pm9\" to finish creating: GET https://rp.j0659584.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9b308276-5a29-4bdc-9917-0a6f5eb47ab9\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9b308276-5a29-4bdc-9917-0a6f5eb47ab9\",\n \"name\": \"9b308276-5a29-4bdc-9917-0a6f5eb47ab9\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-25T23:38:05.329111798Z\",\n \"endTime\": \"2026-02-25T23:40:46.498452335Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj0659584.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:5b666743-501e-0081-38b0-a62840000000\\nTime:2026-02-25T23:40:12.7324517Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n---------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapError \| 0xc000dd6360>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-bz5pm9" to finish creating: GET https://rp.j0659584.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9b308276-5a29-4bdc-9917-0a6f5eb47ab9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9b308276-5a29-4bdc-9917-0a6f5eb47ab9", "name": "9b308276-5a29-4bdc-9917-0a6f5eb47ab9", "status": "Failed", "startTime": "2026-02-25T23:38:05.329111798Z", "endTime": "2026-02-25T23:40:46.498452335Z", "error": { "code": "InternalServerError", "message": "GET https://arohcpprowoidcj0659584.blob.core.windows.net/$web\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\nERROR CODE: AuthorizationPermissionMismatch\n--------------------------------------------------------------------------------\n\u003c?xml version=\"1.0\" encoding=\"utf-8\"?\u003e\u003cError\u003e\u003cCode\u003eAuthorizationPermissionMismatch\u003c/Code\u003e\u003cMessage\u003eThis request is not authorized to perform this operation using this permission.\nRequestId:5b666743-501e-0081-38b0-a62840000000\nTime:2026-02-25T23:40:12.7324517Z\u003c/Message\u003e\u003c/Error\u003e\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-bz5pm9\" to finish creating: GET https://rp.j0659584.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9b308276-5a29-4bdc-9917-0a6f5eb47ab9\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9b308276-5a29-4bdc-9917-0a6f5eb47ab9\",\n \"name\": \"9b308276-5a29-4bdc-9917-0a6f5eb47ab9\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-25T23:38:05.329111798Z\",\n \"endTime\": \"2026-02-25T23:40:46.498452335Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj0659584.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:5b666743-501e-0081-38b0-a62840000000\\nTime:2026-02-25T23:40:12.7324517Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000dd6340>{ msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-bz5pm9\" to finish creating: GET https://rp.j0659584.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9b308276-5a29-4bdc-9917-0a6f5eb47ab9\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9b308276-5a29-4bdc-9917-0a6f5eb47ab9\",\n \"name\": \"9b308276-5a29-4bdc-9917-0a6f5eb47ab9\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-25T23:38:05.329111798Z\",\n \"endTime\": \"2026-02-25T23:40:46.498452335Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"GET https://arohcpprowoidcj0659584.blob.core.windows.net/$web\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 This request is not authorized to perform this operation using this permission.\\nERROR CODE: AuthorizationPermissionMismatch\\n--------------------------------------------------------------------------------\\n\ufeff\\u003c?xml version=\\\"1.0\\\" encoding=\\\"utf-8\\\"?\\u003e\\u003cError\\u003e\\u003cCode\\u003eAuthorizationPermissionMismatch\\u003c/Code\\u003e\\u003cMessage\\u003eThis request is not authorized to perform this operation using this permission.\\nRequestId:5b666743-501e-0081-38b0-a62840000000\\nTime:2026-02-25T23:40:12.7324517Z\\u003c/Message\\u003e\\u003c/Error\\u003e\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n---------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
22:20:15	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c8160d3	n/a	e2e	failed	⚠#4208 (closed)	15	e2e (15) timeout during CreateHCPClusterAndWait; context deadline exceeded 22:20:15 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapErrors \| 0xc000f783f0>: failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-nb4hmc, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-nb4hmc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-nb4hmc, error: context deadline exceeded { msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-nb4hmc, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-nb4hmc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-nb4hmc, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000d1dae0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-nb4hmc", }, <fmt.wrapErrors \| 0xc000f783c0>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-nb4hmc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-nb4hmc, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000d1dae0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-nb4hmc", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapErrors \| 0xc000f783f0>: failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-nb4hmc, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-nb4hmc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-nb4hmc, error: context deadline exceeded { msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-nb4hmc, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-nb4hmc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-nb4hmc, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000d1dae0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-nb4hmc", }, <fmt.wrapErrors \| 0xc000f783c0>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-nb4hmc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-nb4hmc, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000d1dae0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-nb4hmc", }, <context.deadlineExceededError>{}, ], }, ], } occurred Deserializaion Error: no output from command 22:20:15 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure > github.com/Azure/ARO-HCP/test/e2e.init.func15.2({0x7f08e4594240, 0xc0009c44e0}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126 \| \| By("waiting for first cluster to complete creation") > _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{ \| Frequency: framework.StandardPollInterval, \| }) github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8076598?, 0xc0009c44e0?}) /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 github.com/onsi/ginkgo/v2/internal.(Suite).runNode.func3() /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 github.com/onsi/ginkgo/v2/internal.(Suite).runNode in goroutine 111 /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 ------------------------------ [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 02/26/26 00:45:19.484 "ts"="2026-02-26 00:45:19.484354" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-02-26 00:45:19.484434" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-vc4bkz" "resourceGroup"="customer-rg-lkw59k" "ts"="2026-02-26 00:45:20.125570" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-vc4bkz" "resourceGroup"="" "ts"="2026-02-26 00:45:20.656362" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-khzg2s" "resourceGroup"="customer-rg-lkw59k" "ts"="2026-02-26 00:45:20.813594" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-khzg2s" "resourceGroup"="" "ts"="2026-02-26 00:45:21.315441" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz" "ts"="2026-02-26 00:45:21.315574" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz" "ts"="2026-02-26 00:45:21.315650" "level"=0 "msg"="collecting debug info" "ts"="2026-02-26 00:45:21.316740" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-165" "ts"="2026-02-26 00:45:21.316755" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-211" "ts"="2026-02-26 00:45:21.316780" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-lkw59k" "ts"="2026-02-26 00:45:21.316816" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-112" "ts"="2026-02-26 00:45:21.945252" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-165" "deployment"="pooledMsiScopedAssignments" "ts"="2026-02-26 00:45:22.307425" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-211" "deployment"="pooledMsiScopedAssignments" "ts"="2026-02-26 00:45:22.837155" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-lkw59k" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32" "ts"="2026-02-26 00:45:23.153215" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-112" "deployment"="pooledMsiScopedAssignments" "ts"="2026-02-26 00:45:23.589385" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-165" "deployment"="msi-deployment-165" "ts"="2026-02-26 00:45:23.950013" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-211" "deployment"="msi-deployment-211" "ts"="2026-02-26 00:45:24.408245" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-lkw59k" "deployment"="customer-infra-basic-hcp-cluster2-khzg2s" "ts"="2026-02-26 00:45:24.819953" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-112" "deployment"="msi-deployment-112" "ts"="2026-02-26 00:45:26.350566" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-lkw59k" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4" "ts"="2026-02-26 00:45:26.992581" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-lkw59k" "deployment"="customer-infra-basic-hcp-cluster-vc4bkz" "ts"="2026-02-26 00:45:27.598895" "level"=0 "msg"="finished collecting debug info" "ts"="2026-02-26 00:45:27.599053" "level"=0 "msg"="deleting created resources" "ts"="2026-02-26 00:45:27.599173" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-lkw59k" Command Error: signal: killed Deserializaion Error: no output from command > github.com/Azure/ARO-HCP/test/e2e.init.func15.2({0x7f08e4594240, 0xc0009c44e0}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126 \| \| By("waiting for first cluster to complete creation") > _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{ \| Frequency: framework.StandardPollInterval, \| }) github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8076598?, 0xc0009c44e0?}) /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 github.com/onsi/ginkgo/v2/internal.(Suite).runNode.func3() /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 github.com/onsi/ginkgo/v2/internal.(Suite).runNode in goroutine 111 /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 ------------------------------ [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 02/26/26 00:45:19.484 "ts"="2026-02-26 00:45:19.484354" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-02-26 00:45:19.484434" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-vc4bkz" "resourceGroup"="customer-rg-lkw59k" "ts"="2026-02-26 00:45:20.125570" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-vc4bkz" "resourceGroup"="" "ts"="2026-02-26 00:45:20.656362" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-khzg2s" "resourceGroup"="customer-rg-lkw59k" "ts"="2026-02-26 00:45:20.813594" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-khzg2s" "resourceGroup"="" "ts"="2026-02-26 00:45:21.315441" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz" "ts"="2026-02-26 00:45:21.315574" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz" "ts"="2026-02-26 00:45:21.315650" "level"=0 "msg"="collecting debug info" "ts"="2026-02-26 00:45:21.316740" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-165" "ts"="2026-02-26 00:45:21.316755" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-211" "ts"="2026-02-26 00:45:21.316780" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-lkw59k" "ts"="2026-02-26 00:45:21.316816" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-112" "ts"="2026-02-26 00:45:21.945252" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-165" "deployment"="pooledMsiScopedAssignments" "ts"="2026-02-26 00:45:22.307425" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-211" "deployment"="pooledMsiScopedAssignments" "ts"="2026-02-26 00:45:22.837155" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-lkw59k" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32" "ts"="2026-02-26 00:45:23.153215" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-112" "deployment"="pooledMsiScopedAssignments" "ts"="2026-02-26 00:45:23.589385" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-165" "deployment"="msi-deployment-165" "ts"="2026-02-26 00:45:23.950013" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-211" "deployment"="msi-deployment-211" "ts"="2026-02-26 00:45:24.408245" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-lkw59k" "deployment"="customer-infra-basic-hcp-cluster2-khzg2s" "ts"="2026-02-26 00:45:24.819953" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-112" "deployment"="msi-deployment-112" "ts"="2026-02-26 00:45:26.350566" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-lkw59k" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4" "ts"="2026-02-26 00:45:26.992581" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-lkw59k" "deployment"="customer-infra-basic-hcp-cluster-vc4bkz" "ts"="2026-02-26 00:45:27.598895" "level"=0 "msg"="finished collecting debug info" "ts"="2026-02-26 00:45:27.599053" "level"=0 "msg"="deleting created resources" "ts"="2026-02-26 00:45:27.599173" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-lkw59k" Command Error: signal: killed Deserializaion Error: no output from command timeout during CreateHCPClusterAndWait; context deadline exceeded 22:20:15 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error: <fmt.wrapErrors \| 0xc0010ec690>: failed to create HCP cluster basic-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-q5thkp, error: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-q5thkp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-q5thkp, error: context deadline exceeded { msg: "failed to create HCP cluster basic-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-q5thkp, error: failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-q5thkp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-q5thkp, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0010af610>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-q5thkp", }, <fmt.wrapErrors \| 0xc0010ec660>{ msg: "failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-q5thkp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-q5thkp, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0010af610>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-q5thkp", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error: <fmt.wrapErrors \| 0xc0010ec690>: failed to create HCP cluster basic-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-q5thkp, error: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-q5thkp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-q5thkp, error: context deadline exceeded { msg: "failed to create HCP cluster basic-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-q5thkp, error: failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-q5thkp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-q5thkp, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0010af610>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-q5thkp", }, <fmt.wrapErrors \| 0xc0010ec660>{ msg: "failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-q5thkp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-q5thkp, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0010af610>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-q5thkp", }, <context.deadlineExceededError>{}, ], }, ], } occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 22:20:15 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapErrors \| 0xc00155b3e0>: failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-7dwxjz, error: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-7dwxjz" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-7dwxjz, error: context deadline exceeded { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-7dwxjz, error: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-7dwxjz\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-7dwxjz, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000dadd40>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-7dwxjz", }, <fmt.wrapErrors \| 0xc00155b3b0>{ msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-7dwxjz\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-7dwxjz, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000dadd40>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-7dwxjz", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapErrors \| 0xc00155b3e0>: failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-7dwxjz, error: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-7dwxjz" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-7dwxjz, error: context deadline exceeded { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-7dwxjz, error: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-7dwxjz\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-7dwxjz, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000dadd40>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-7dwxjz", }, <fmt.wrapErrors \| 0xc00155b3b0>{ msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-7dwxjz\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-7dwxjz, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000dadd40>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-7dwxjz", }, <context.deadlineExceededError>{}, ], }, ], } occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 22:20:15 UTC · failed at e2e · test Customer should be able to list HCP clusters without node pools at both subscription and resource group levels · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error: <fmt.wrapErrors \| 0xc001167770>: failed to create HCP cluster list-test-cluster-2z4qgp, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-2z4qgp in resource group cluster-listing-qttmzq, error: failed waiting for cluster="list-test-cluster-2z4qgp" in resourcegroup="cluster-listing-qttmzq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-2z4qgp in resource group cluster-listing-qttmzq, error: context deadline exceeded { msg: "failed to create HCP cluster list-test-cluster-2z4qgp, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-2z4qgp in resource group cluster-listing-qttmzq, error: failed waiting for cluster=\"list-test-cluster-2z4qgp\" in resourcegroup=\"cluster-listing-qttmzq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-2z4qgp in resource group cluster-listing-qttmzq, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001150800>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-2z4qgp in resource group cluster-listing-qttmzq", }, <fmt.wrapErrors \| 0xc001167740>{ msg: "failed waiting for cluster=\"list-test-cluster-2z4qgp\" in resourcegroup=\"cluster-listing-qttmzq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-2z4qgp in resource group cluster-listing-qttmzq, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001150800>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-2z4qgp in resource group cluster-listing-qttmzq", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error: <fmt.wrapErrors \| 0xc001167770>: failed to create HCP cluster list-test-cluster-2z4qgp, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-2z4qgp in resource group cluster-listing-qttmzq, error: failed waiting for cluster="list-test-cluster-2z4qgp" in resourcegroup="cluster-listing-qttmzq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-2z4qgp in resource group cluster-listing-qttmzq, error: context deadline exceeded { msg: "failed to create HCP cluster list-test-cluster-2z4qgp, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-2z4qgp in resource group cluster-listing-qttmzq, error: failed waiting for cluster=\"list-test-cluster-2z4qgp\" in resourcegroup=\"cluster-listing-qttmzq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-2z4qgp in resource group cluster-listing-qttmzq, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001150800>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-2z4qgp in resource group cluster-listing-qttmzq", }, <fmt.wrapErrors \| 0xc001167740>{ msg: "failed waiting for cluster=\"list-test-cluster-2z4qgp\" in resourcegroup=\"cluster-listing-qttmzq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-2z4qgp in resource group cluster-listing-qttmzq, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001150800>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-2z4qgp in resource group cluster-listing-qttmzq", }, <context.deadlineExceededError>{}, ], }, ], } occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 22:20:15 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error: <fmt.wrapErrors \| 0xc000fd8870>: failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-jg87w4, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-jg87w4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-jg87w4, error: context deadline exceeded { msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-jg87w4, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-jg87w4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-jg87w4, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000975040>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-jg87w4", }, <fmt.wrapErrors \| 0xc000fd8840>{ msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-jg87w4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-jg87w4, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000975040>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-jg87w4", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error: <fmt.wrapErrors \| 0xc000fd8870>: failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-jg87w4, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-jg87w4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-jg87w4, error: context deadline exceeded { msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-jg87w4, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-jg87w4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-jg87w4, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000975040>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-jg87w4", }, <fmt.wrapErrors \| 0xc000fd8840>{ msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-jg87w4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-jg87w4, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000975040>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-jg87w4", }, <context.deadlineExceededError>{}, ], }, ], } occurred Interrupted by User 22:20:15 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [:0]: Interrupted by User fail [:0]: Interrupted by User timeout during CreateHCPClusterAndWait; context deadline exceeded 22:20:15 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error: <fmt.wrapErrors \| 0xc000c44c60>: failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-dxmbbp, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-dxmbbp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-dxmbbp, error: context deadline exceeded { msg: "failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-dxmbbp, error: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-dxmbbp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-dxmbbp, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0006a0fa0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-dxmbbp", }, <fmt.wrapErrors \| 0xc000c44c30>{ msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-dxmbbp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-dxmbbp, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0006a0fa0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-dxmbbp", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error: <fmt.wrapErrors \| 0xc000c44c60>: failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-dxmbbp, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-dxmbbp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-dxmbbp, error: context deadline exceeded { msg: "failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-dxmbbp, error: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-dxmbbp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-dxmbbp, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0006a0fa0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-dxmbbp", }, <fmt.wrapErrors \| 0xc000c44c30>{ msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-dxmbbp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-dxmbbp, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0006a0fa0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-dxmbbp", }, <context.deadlineExceededError>{}, ], }, ], } occurred Cluster should become ready within 45 minutes 22:20:15 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s. Cluster should become ready within 45 minutes Expected <bool>: false to be true fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s. Cluster should become ready within 45 minutes Expected <bool>: false to be true timeout during CreateHCPClusterAndWait; context deadline exceeded 22:20:15 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapErrors \| 0xc0011d0d80>: failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-bwmbmd, error: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-bwmbmd" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-bwmbmd, error: context deadline exceeded { msg: "failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-bwmbmd, error: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-bwmbmd\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-bwmbmd, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001309a10>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-bwmbmd", }, <fmt.wrapErrors \| 0xc0011d0d50>{ msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-bwmbmd\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-bwmbmd, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001309a10>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-bwmbmd", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapErrors \| 0xc0011d0d80>: failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-bwmbmd, error: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-bwmbmd" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-bwmbmd, error: context deadline exceeded { msg: "failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-bwmbmd, error: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-bwmbmd\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-bwmbmd, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001309a10>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-bwmbmd", }, <fmt.wrapErrors \| 0xc0011d0d50>{ msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-bwmbmd\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-bwmbmd, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001309a10>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-bwmbmd", }, <context.deadlineExceededError>{}, ], }, ], } occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 22:20:15 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error: <fmt.wrapErrors \| 0xc000a4e420>: failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-qtsmtv, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-qtsmtv" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-qtsmtv, error: context deadline exceeded { msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-qtsmtv, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-qtsmtv\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-qtsmtv, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0008bc0e0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-qtsmtv", }, <fmt.wrapErrors \| 0xc000a4e3f0>{ msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-qtsmtv\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-qtsmtv, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0008bc0e0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-qtsmtv", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error: <fmt.wrapErrors \| 0xc000a4e420>: failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-qtsmtv, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-qtsmtv" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-qtsmtv, error: context deadline exceeded { msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-qtsmtv, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-qtsmtv\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-qtsmtv, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0008bc0e0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-qtsmtv", }, <fmt.wrapErrors \| 0xc000a4e3f0>{ msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-qtsmtv\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-qtsmtv, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0008bc0e0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-qtsmtv", }, <context.deadlineExceededError>{}, ], }, ], } occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 22:20:15 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapErrors \| 0xc000ecd5f0>: failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-pzm49x, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-pzm49x" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-pzm49x, error: context deadline exceeded { msg: "failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-pzm49x, error: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-pzm49x\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-pzm49x, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00117e010>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-pzm49x", }, <fmt.wrapErrors \| 0xc000ecd5c0>{ msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-pzm49x\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-pzm49x, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00117e010>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-pzm49x", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapErrors \| 0xc000ecd5f0>: failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-pzm49x, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-pzm49x" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-pzm49x, error: context deadline exceeded { msg: "failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-pzm49x, error: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-pzm49x\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-pzm49x, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00117e010>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-pzm49x", }, <fmt.wrapErrors \| 0xc000ecd5c0>{ msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-pzm49x\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-pzm49x, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00117e010>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-pzm49x", }, <context.deadlineExceededError>{}, ], }, ], } occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 22:20:15 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:90]: Unexpected error: <fmt.wrapErrors \| 0xc0015fc6c0>: failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-5kgdld, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-5kgdld" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-5kgdld, error: context deadline exceeded { msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-5kgdld, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5kgdld\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-5kgdld, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001720cd0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-5kgdld", }, <fmt.wrapErrors \| 0xc0015fc690>{ msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5kgdld\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-5kgdld, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001720cd0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-5kgdld", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:90]: Unexpected error: <fmt.wrapErrors \| 0xc0015fc6c0>: failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-5kgdld, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-5kgdld" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-5kgdld, error: context deadline exceeded { msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-5kgdld, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5kgdld\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-5kgdld, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001720cd0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-5kgdld", }, <fmt.wrapErrors \| 0xc0015fc690>{ msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5kgdld\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-5kgdld, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001720cd0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-5kgdld", }, <context.deadlineExceededError>{}, ], }, ], } occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 22:20:15 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:100]: Unexpected error: <fmt.wrapErrors \| 0xc0011669f0>: failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-b9dtbr, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-b9dtbr" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-b9dtbr, error: context deadline exceeded { msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-b9dtbr, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-b9dtbr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-b9dtbr, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00115e360>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-b9dtbr", }, <fmt.wrapErrors \| 0xc0011669c0>{ msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-b9dtbr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-b9dtbr, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00115e360>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-b9dtbr", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:100]: Unexpected error: <fmt.wrapErrors \| 0xc0011669f0>: failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-b9dtbr, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-b9dtbr" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-b9dtbr, error: context deadline exceeded { msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-b9dtbr, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-b9dtbr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-b9dtbr, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00115e360>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-b9dtbr", }, <fmt.wrapErrors \| 0xc0011669c0>{ msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-b9dtbr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-b9dtbr, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00115e360>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-b9dtbr", }, <context.deadlineExceededError>{}, ], }, ], } occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 22:20:15 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error: <fmt.wrapErrors \| 0xc000d5b350>: failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-b6zr97, error: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-b6zr97" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-b6zr97, error: context deadline exceeded { msg: "failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-b6zr97, error: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-b6zr97\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-b6zr97, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000f67b00>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-b6zr97", }, <fmt.wrapErrors \| 0xc000d5b320>{ msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-b6zr97\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-b6zr97, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000f67b00>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-b6zr97", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error: <fmt.wrapErrors \| 0xc000d5b350>: failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-b6zr97, error: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-b6zr97" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-b6zr97, error: context deadline exceeded { msg: "failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-b6zr97, error: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-b6zr97\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-b6zr97, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000f67b00>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-b6zr97", }, <fmt.wrapErrors \| 0xc000d5b320>{ msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-b6zr97\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-b6zr97, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000f67b00>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-b6zr97", }, <context.deadlineExceededError>{}, ], }, ], } occurred
22:17:19	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 3ca804d · merge 3ca804d	n/a	provision	failed	★#3679 (merged)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 22:17:19 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-azure-plugin-image · suite step graph Show raw failure time=2026-02-25T22:37:27.977Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image description="Step mirror-azure-plugin-image\n Kind: ImageMirror\n From {<nil> velero.azurePlugin.registry <nil>}:{<nil> velero.azurePlugin.repository <nil>}@{<nil> velero.azurePlugin.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T22:37:30.343Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9@sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e to arohcpsvcdev.azurecr.io/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e.\nThe image will still be available under it's original digest sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e in the target registry.\nError response from registry: failed to resolve sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e: GET \"https://registry.redhat.io/v2/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9/manifests/sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
22:02:32	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 551e259 · merge 551e259	n/a	provision	failed	★#4212 (merged)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 22:02:32 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-hypershift-plugin-image · suite step graph Show raw failure time=2026-02-25T22:44:52.734Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-hypershift-plugin-image description="Step mirror-hypershift-plugin-image\n Kind: ImageMirror\n From {<nil> velero.hypershiftPlugin.registry <nil>}:{<nil> velero.hypershiftPlugin.repository <nil>}@{<nil> velero.hypershiftPlugin.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T22:44:55.660Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-hypershift-plugin-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e to arohcpsvcdev.azurecr.io/oadp/oadp-hypershift-velero-plugin-rhel9:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e.\nThe image will still be available under it's original digest sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e in the target registry.\nError response from registry: failed to resolve sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e: GET \"https://registry.redhat.io/v2/oadp/oadp-hypershift-velero-plugin-rhel9/manifests/sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
21:43:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head e9d3aed · merge e9d3aed	n/a	provision	failed	★#4216 (merged)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 21:43:11 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-hypershift-plugin-image · suite step graph Show raw failure time=2026-02-25T22:27:52.583Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-hypershift-plugin-image description="Step mirror-hypershift-plugin-image\n Kind: ImageMirror\n From {<nil> velero.hypershiftPlugin.registry <nil>}:{<nil> velero.hypershiftPlugin.repository <nil>}@{<nil> velero.hypershiftPlugin.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T22:27:55.317Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-hypershift-plugin-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e to arohcpsvcdev.azurecr.io/oadp/oadp-hypershift-velero-plugin-rhel9:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e.\nThe image will still be available under it's original digest sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e in the target registry.\nError response from registry: failed to resolve sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e: GET \"https://registry.redhat.io/v2/oadp/oadp-hypershift-velero-plugin-rhel9/manifests/sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
21:19:14	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 3ca804d · merge 3ca804d	n/a	provision	failed	★#3679 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid 21:19:14 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph Show raw failure time=2026-02-25T21:36:37.127Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-02-25T21:36:39.407Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra time=2026-02-25T21:36:40.831Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=989935f8672b1a8791202138e4b994003547af7139d0aa916f5d7417cf6d6f38 time=2026-02-25T21:38:11.597Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j7069440/providers/Microsoft.Resources/deployments/989935f8672b1a8791202138e4b994003547af7139d0aa916f5d7417cf6d6f38/operationStatuses/08584295514854648601\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7069440/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7069440/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j7069440\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Internal error\\\",\\r\\n \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 0ca16638-1521-4555-b306-e0b991d316b6:2/25/2026 9:37:57 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
20:37:55	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head dcf56c7 · merge dcf56c7	n/a	provision	failed	★#3780 (merged)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 20:37:55 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-azure-plugin-image · suite step graph Show raw failure time=2026-02-25T21:29:36.893Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image description="Step mirror-azure-plugin-image\n Kind: ImageMirror\n From {<nil> velero.azurePlugin.registry <nil>}:{<nil> velero.azurePlugin.repository <nil>}@{<nil> velero.azurePlugin.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T21:29:39.584Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9@sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e to arohcpsvcdev.azurecr.io/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e.\nThe image will still be available under it's original digest sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e in the target registry.\nError response from registry: failed to resolve sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e: GET \"https://registry.redhat.io/v2/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9/manifests/sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
20:35:26	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c8160d3	n/a	provision	failed	★#4208 (closed)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 20:35:26 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-hypershift-plugin-image · suite step graph Show raw failure time=2026-02-25T21:20:24.336Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-hypershift-plugin-image description="Step mirror-hypershift-plugin-image\n Kind: ImageMirror\n From {<nil> velero.hypershiftPlugin.registry <nil>}:{<nil> velero.hypershiftPlugin.repository <nil>}@{<nil> velero.hypershiftPlugin.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T21:20:26.980Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-hypershift-plugin-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e to arohcpsvcdev.azurecr.io/oadp/oadp-hypershift-velero-plugin-rhel9:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e.\nThe image will still be available under it's original digest sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e in the target registry.\nError response from registry: failed to resolve sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e: GET \"https://registry.redhat.io/v2/oadp/oadp-hypershift-velero-plugin-rhel9/manifests/sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
20:23:56	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 096bc31 · merge 096bc31	n/a	provision	failed	★#4213 (merged)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 20:23:56 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-azure-plugin-image · suite step graph Show raw failure time=2026-02-25T21:12:48.575Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image description="Step mirror-azure-plugin-image\n Kind: ImageMirror\n From {<nil> velero.azurePlugin.registry <nil>}:{<nil> velero.azurePlugin.repository <nil>}@{<nil> velero.azurePlugin.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T21:12:50.788Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9@sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e to arohcpsvcdev.azurecr.io/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e.\nThe image will still be available under it's original digest sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e in the target registry.\nError response from registry: failed to resolve sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e: GET \"https://registry.redhat.io/v2/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9/manifests/sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
20:10:34	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 8e248ec	n/a	provision	failed	★#4211 (open)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 20:10:34 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-azure-plugin-image · suite step graph Show raw failure time=2026-02-25T20:58:33.207Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image description="Step mirror-azure-plugin-image\n Kind: ImageMirror\n From {<nil> velero.azurePlugin.registry <nil>}:{<nil> velero.azurePlugin.repository <nil>}@{<nil> velero.azurePlugin.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T20:58:35.555Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9@sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e to arohcpsvcdev.azurecr.io/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e.\nThe image will still be available under it's original digest sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e in the target registry.\nError response from registry: failed to resolve sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e: GET \"https://registry.redhat.io/v2/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9/manifests/sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
19:58:22	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 3ca804d · merge 3ca804d	n/a	provision	failed	★#3679 (merged)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 19:58:22 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-velero-image · suite step graph Show raw failure time=2026-02-25T20:59:12.201Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image description="Step mirror-velero-image\n Kind: ImageMirror\n From {<nil> velero.server.registry <nil>}:{<nil> velero.server.repository <nil>}@{<nil> velero.server.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T20:59:14.874Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-velero-rhel9@sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b to arohcpsvcdev.azurecr.io/oadp/oadp-velero-rhel9:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b.\nThe image will still be available under it's original digest sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b in the target registry.\nError response from registry: failed to resolve sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b: GET \"https://registry.redhat.io/v2/oadp/oadp-velero-rhel9/manifests/sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
19:39:46	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 395fd83 · merge 551e259	n/a	provision	failed	⚠#4212 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code DeploymentPreflightValidationFailed; detail message The script language version '15.3.0' is not supported. Supported versions: '2.7,2.8,3.0,3.1,3.2,3.3,3.4,3.5,3.6,3.7,3.8,4.1,4.2,4.3,4.4,4.5,4.6,4.7,4.8,5.0,5.1,5.2,5.3,5.4,5.5,5.6,5.7,5.8,5.9,6.0,6.1,6.2,6.3,6.4,6.5,6.6,7.0,7.1,7.2,7.3,7.4,7.5,8.0,8.1,8.2,8.3,9.0,9.1,9.2,9.3,9.4,9.5,9.6,9.7,10.0,10.1,10.2,10.3,10.4,11.0,11.1,11.2,11.3,11.4,11.5,11.6,12.0,12.1,12.2,12.3,12.4,12.5,13.0,13.1,13.2,13.3,13.4,13.5,14.0,14.1,14.2,14.3,14.4,14.5,14.6,15.0,15.1'. Please 19:39:46 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-02-25T20:35:16.832Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-02-25T20:35:18.519Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-02-25T20:35:20.565Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=26b9c29583a7924ad3a98fccd321c157eec0f1139ce2debf0027ac3e6c51faca time=2026-02-25T20:50:24.199Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j2736256-svc/providers/Microsoft.Resources/deployments/26b9c29583a7924ad3a98fccd321c157eec0f1139ce2debf0027ac3e6c51faca/operationStatuses/08584295551659603455\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2736256-svc/providers/Microsoft.Resources/deployments/maestro-server\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/maestro-eg-crt-rcvcdydzmzklu\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/maestro-eg-crt-rcvcdydzmzklu\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"InvalidTemplateDeployment\\\",\\r\\n \\\"message\\\": \\\"The template deployment 'maestro-server-j2736256-cert' is not valid according to the validation procedure. The tracking id is '437861dd-327e-4979-892c-30ec68200dc1'. See inner errors for details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentPreflightValidationFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deploymentScripts/newCertwithRotationKV-maestro-server-j2736256\\\",\\r\\n \\\"message\\\": \\\"The script language version '15.3.0' is not supported. Supported versions: '2.7,2.8,3.0,3.1,3.2,3.3,3.4,3.5,3.6,3.7,3.8,4.1,4.2,4.3,4.4,4.5,4.6,4.7,4.8,5.0,5.1,5.2,5.3,5.4,5.5,5.6,5.7,5.8,5.9,6.0,6.1,6.2,6.3,6.4,6.5,6.6,7.0,7.1,7.2,7.3,7.4,7.5,8.0,8.1,8.2,8.3,9.0,9.1,9.2,9.3,9.4,9.5,9.6,9.7,10.0,10.1,10.2,10.3,10.4,11.0,11.1,11.2,11.3,11.4,11.5,11.6,12.0,12.1,12.2,12.3,12.4,12.5,13.0,13.1,13.2,13.3,13.4,13.5,14.0,14.1,14.2,14.3,14.4,14.5,14.6,15.0,15.1'. Please refer to https://aka.ms/DeploymentScriptsTroubleshoot for more deployment script information.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"InvalidTemplateDeployment\\\",\\r\\n \\\"message\\\": \\\"The template deployment 'frontend-cert-uxl4bs2oechw4' is not valid according to the validation procedure. The tracking id is 'cfe19b22-d56a-4e05-ab80-54b3c9aba2a9'. See inner errors for details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentPreflightValidationFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deploymentScripts/newCertwithRotationKV-frontend-cert-prow-j2736256\\\",\\r\\n \\\"message\\\": \\\"The script language version '15.3.0' is not supported. Supported versions: '2.7,2.8,3.0,3.1,3.2,3.3,3.4,3.5,3.6,3.7,3.8,4.1,4.2,4.3,4.4,4.5,4.6,4.7,4.8,5.0,5.1,5.2,5.3,5.4,5.5,5.6,5.7,5.8,5.9,6.0,6.1,6.2,6.3,6.4,6.5,6.6,7.0,7.1,7.2,7.3,7.4,7.5,8.0,8.1,8.2,8.3,9.0,9.1,9.2,9.3,9.4,9.5,9.6,9.7,10.0,10.1,10.2,10.3,10.4,11.0,11.1,11.2,11.3,11.4,11.5,11.6,12.0,12.1,12.2,12.3,12.4,12.5,13.0,13.1,13.2,13.3,13.4,13.5,14.0,14.1,14.2,14.3,14.4,14.5,14.6,15.0,15.1'. Please refer to https://aka.ms/DeploymentScriptsTroubleshoot for more deployment script information.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"InvalidTemplateDeployment\\\",\\r\\n \\\"message\\\": \\\"The template deployment 'admin-api-cert-uxl4bs2oechw4' is not valid according to the validation procedure. The tracking id is '3c53241c-9cfb-4e71-8f46-0fe5ea8a07f6'. See inner errors for details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentPreflightValidationFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deploymentScripts/newCertwithRotationKV-admin-api-cert-prow-j2736256\\\",\\r\\n \\\"message\\\": \\\"The script language version '15.3.0' is not supported. Supported versions: '2.7,2.8,3.0,3.1,3.2,3.3,3.4,3.5,3.6,3.7,3.8,4.1,4.2,4.3,4.4,4.5,4.6,4.7,4.8,5.0,5.1,5.2,5.3,5.4,5.5,5.6,5.7,5.8,5.9,6.0,6.1,6.2,6.3,6.4,6.5,6.6,7.0,7.1,7.2,7.3,7.4,7.5,8.0,8.1,8.2,8.3,9.0,9.1,9.2,9.3,9.4,9.5,9.6,9.7,10.0,10.1,10.2,10.3,10.4,11.0,11.1,11.2,11.3,11.4,11.5,11.6,12.0,12.1,12.2,12.3,12.4,12.5,13.0,13.1,13.2,13.3,13.4,13.5,14.0,14.1,14.2,14.3,14.4,14.5,14.6,15.0,15.1'. Please refer to https://aka.ms/DeploymentScriptsTroubleshoot for more deployment script information.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
19:39:37	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 395fd83 · merge 551e259	n/a	unknown	failed	#4212 (merged)	1	non-artifact-backed failure (no junit artifacts)
19:26:05	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 79f03ff	n/a	provision	failed	★#4033 (closed)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 19:26:05 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-velero-image · suite step graph Show raw failure time=2026-02-25T20:18:13.404Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image description="Step mirror-velero-image\n Kind: ImageMirror\n From {<nil> velero.server.registry <nil>}:{<nil> velero.server.repository <nil>}@{<nil> velero.server.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T20:18:16.373Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-velero-rhel9@sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b to arohcpsvcdev.azurecr.io/oadp/oadp-velero-rhel9:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b.\nThe image will still be available under it's original digest sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b in the target registry.\nError response from registry: failed to resolve sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b: GET \"https://registry.redhat.io/v2/oadp/oadp-velero-rhel9/manifests/sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
19:17:31	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head b1cdfeb · merge b1cdfeb	n/a	provision	failed	★#4206 (merged)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 19:17:31 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-hypershift-plugin-image · suite step graph Show raw failure time=2026-02-25T20:14:00.044Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-hypershift-plugin-image description="Step mirror-hypershift-plugin-image\n Kind: ImageMirror\n From {<nil> velero.hypershiftPlugin.registry <nil>}:{<nil> velero.hypershiftPlugin.repository <nil>}@{<nil> velero.hypershiftPlugin.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T20:14:04.485Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-hypershift-plugin-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e to arohcpsvcdev.azurecr.io/oadp/oadp-hypershift-velero-plugin-rhel9:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e.\nThe image will still be available under it's original digest sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e in the target registry.\nError response from registry: failed to resolve sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e: GET \"https://registry.redhat.io/v2/oadp/oadp-hypershift-velero-plugin-rhel9/manifests/sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
19:16:49	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4bea39e	n/a	provision	failed	★#3094 (closed)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 19:16:49 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-azure-plugin-image · suite step graph Show raw failure time=2026-02-25T20:19:25.705Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image description="Step mirror-azure-plugin-image\n Kind: ImageMirror\n From {<nil> velero.azurePlugin.registry <nil>}:{<nil> velero.azurePlugin.repository <nil>}@{<nil> velero.azurePlugin.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T20:19:28.729Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9@sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e to arohcpsvcdev.azurecr.io/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e.\nThe image will still be available under it's original digest sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e in the target registry.\nError response from registry: failed to resolve sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e: GET \"https://registry.redhat.io/v2/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9/manifests/sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
19:10:33	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head a02b279 · merge a02b279	n/a	provision	failed	★#4207 (merged)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 19:10:33 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-velero-image · suite step graph Show raw failure time=2026-02-25T20:04:24.765Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image description="Step mirror-velero-image\n Kind: ImageMirror\n From {<nil> velero.server.registry <nil>}:{<nil> velero.server.repository <nil>}@{<nil> velero.server.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T20:04:27.592Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-velero-rhel9@sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b to arohcpsvcdev.azurecr.io/oadp/oadp-velero-rhel9:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b.\nThe image will still be available under it's original digest sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b in the target registry.\nError response from registry: failed to resolve sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b: GET \"https://registry.redhat.io/v2/oadp/oadp-velero-rhel9/manifests/sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
18:55:27	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 60d3a82 · merge 096bc31	n/a	provision	failed	★#4213 (merged)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 18:55:27 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-azure-plugin-image · suite step graph Show raw failure time=2026-02-25T19:40:13.244Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image description="Step mirror-azure-plugin-image\n Kind: ImageMirror\n From {<nil> velero.azurePlugin.registry <nil>}:{<nil> velero.azurePlugin.repository <nil>}@{<nil> velero.azurePlugin.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T19:40:15.943Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9@sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e to arohcpsvcdev.azurecr.io/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e.\nThe image will still be available under it's original digest sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e in the target registry.\nError response from registry: failed to resolve sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e: GET \"https://registry.redhat.io/v2/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9/manifests/sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
18:28:06	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head b8334fe	n/a	provision	failed	★#4211 (open)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 18:28:06 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-velero-image · suite step graph Show raw failure time=2026-02-25T19:16:56.156Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image description="Step mirror-velero-image\n Kind: ImageMirror\n From {<nil> velero.server.registry <nil>}:{<nil> velero.server.repository <nil>}@{<nil> velero.server.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T19:16:58.639Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-velero-rhel9@sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b to arohcpsvcdev.azurecr.io/oadp/oadp-velero-rhel9:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b.\nThe image will still be available under it's original digest sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b in the target registry.\nError response from registry: failed to resolve sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b: GET \"https://registry.redhat.io/v2/oadp/oadp-velero-rhel9/manifests/sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
18:06:41	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 3ca804d · merge 3ca804d	n/a	provision	failed	★#3679 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code ConcurrentFederatedIdentityCredentialsWritesForSingleManagedIdentity; detail message Too many Federated Identity Credentials are written concurrently for the managed identity '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.managedidentity/userassignedidentities/image-puller'. Concurrent Federated Identity Credentials writes under the same managed identity are not supported. 18:06:41 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster · suite step graph Show raw failure time=2026-02-25T18:47:54.409Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster description="Step cluster\n Kind: ARM\n Template: templates/mgmt-cluster.bicep\n Parameters: configurations/mgmt-cluster.tmpl.bicepparam" time=2026-02-25T18:47:55.212Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster time=2026-02-25T18:47:56.787Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster deployment=f80e6f8b3407980613db5c04e9a30af4573312f0057b17fbc2b8f5eab928520f time=2026-02-25T18:57:29.381Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j7841152-mgmt-1/providers/Microsoft.Resources/deployments/f80e6f8b3407980613db5c04e9a30af4573312f0057b17fbc2b8f5eab928520f/operationStatuses/08584295616096671417\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7841152-mgmt-1/providers/Microsoft.Resources/deployments/cluster-gu7qpftcdii5y\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ConcurrentFederatedIdentityCredentialsWritesForSingleManagedIdentity\\\",\\r\\n \\\"message\\\": \\\"Too many Federated Identity Credentials are written concurrently for the managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j7841152-mgmt-1/providers/microsoft.managedidentity/userassignedidentities/image-puller'. Concurrent Federated Identity Credentials writes under the same managed identity are not supported.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
18:03:24	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4bea39e	n/a	provision	failed	★#3094 (closed)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 18:03:24 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-velero-image · suite step graph Show raw failure time=2026-02-25T18:55:03.525Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image description="Step mirror-velero-image\n Kind: ImageMirror\n From {<nil> velero.server.registry <nil>}:{<nil> velero.server.repository <nil>}@{<nil> velero.server.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T18:55:06.439Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-velero-rhel9@sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b to arohcpsvcdev.azurecr.io/oadp/oadp-velero-rhel9:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b.\nThe image will still be available under it's original digest sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b in the target registry.\nError response from registry: failed to resolve sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b: GET \"https://registry.redhat.io/v2/oadp/oadp-velero-rhel9/manifests/sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
18:00:39	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4bea39e	n/a	provision	failed	★#3094 (closed)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 18:00:39 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-azure-plugin-image · suite step graph Show raw failure time=2026-02-25T18:50:20.927Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image description="Step mirror-azure-plugin-image\n Kind: ImageMirror\n From {<nil> velero.azurePlugin.registry <nil>}:{<nil> velero.azurePlugin.repository <nil>}@{<nil> velero.azurePlugin.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T18:50:23.621Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9@sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e to arohcpsvcdev.azurecr.io/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e.\nThe image will still be available under it's original digest sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e in the target registry.\nError response from registry: failed to resolve sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e: GET \"https://registry.redhat.io/v2/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9/manifests/sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
18:00:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 395fd83 · merge 551e259	n/a	provision	failed	★#4212 (merged)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 18:00:23 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-azure-plugin-image · suite step graph Show raw failure time=2026-02-25T18:50:19.854Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image description="Step mirror-azure-plugin-image\n Kind: ImageMirror\n From {<nil> velero.azurePlugin.registry <nil>}:{<nil> velero.azurePlugin.repository <nil>}@{<nil> velero.azurePlugin.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T18:50:22.789Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9@sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e to arohcpsvcdev.azurecr.io/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e.\nThe image will still be available under it's original digest sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e in the target registry.\nError response from registry: failed to resolve sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e: GET \"https://registry.redhat.io/v2/oadp/oadp-velero-plugin-for-microsoft-azure-rhel9/manifests/sha256:47a1a93a04208748c29073c17855aeca8f72ff2025d7f404b054abd375c8172e\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
17:57:50	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4bea39e	n/a	provision	failed	★#3094 (closed)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 17:57:50 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-hypershift-plugin-image · suite step graph Show raw failure time=2026-02-25T18:55:38.447Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-hypershift-plugin-image description="Step mirror-hypershift-plugin-image\n Kind: ImageMirror\n From {<nil> velero.hypershiftPlugin.registry <nil>}:{<nil> velero.hypershiftPlugin.repository <nil>}@{<nil> velero.hypershiftPlugin.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T18:55:41.126Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-hypershift-plugin-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-hypershift-velero-plugin-rhel9@sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e to arohcpsvcdev.azurecr.io/oadp/oadp-hypershift-velero-plugin-rhel9:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e.\nThe image will still be available under it's original digest sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e in the target registry.\nError response from registry: failed to resolve sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e: GET \"https://registry.redhat.io/v2/oadp/oadp-hypershift-velero-plugin-rhel9/manifests/sha256:f47f2f9099b6ad235ea1187556dc165aef725bcbe2b818ab62bb186ad485961e\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
17:46:50	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0c3f47c	n/a	provision	failed	★#4210 (open)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 17:46:50 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-velero-image · suite step graph Show raw failure time=2026-02-25T18:43:32.187Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image description="Step mirror-velero-image\n Kind: ImageMirror\n From {<nil> velero.server.registry <nil>}:{<nil> velero.server.repository <nil>}@{<nil> velero.server.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T18:43:35.528Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-velero-rhel9@sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b to arohcpsvcdev.azurecr.io/oadp/oadp-velero-rhel9:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b.\nThe image will still be available under it's original digest sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b in the target registry.\nError response from registry: failed to resolve sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b: GET \"https://registry.redhat.io/v2/oadp/oadp-velero-rhel9/manifests/sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
17:38:12	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4e06eac	n/a	provision	failed	★#3925 (open)	1	provision (1) Error response from registry: failed to resolve sha256:<hex>: GET "<url>": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: <url> 17:38:12 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-velero-image · suite step graph Show raw failure time=2026-02-25T18:31:33.569Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image description="Step mirror-velero-image\n Kind: ImageMirror\n From {<nil> velero.server.registry <nil>}:{<nil> velero.server.repository <nil>}@{<nil> velero.server.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T18:31:36.356Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.redhat.io/oadp/oadp-velero-rhel9@sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b to arohcpsvcdev.azurecr.io/oadp/oadp-velero-rhel9:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b.\nThe image will still be available under it's original digest sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b in the target registry.\nError response from registry: failed to resolve sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b: GET \"https://registry.redhat.io/v2/oadp/oadp-velero-rhel9/manifests/sha256:8ee2114096549e96bce5974584d14018c85e69bb8748588a89b960fdf502f64b\": unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n exit status 1"
17:19:16	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 5a03840 · merge b541ca7	n/a	provision	failed	★#4196 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid 17:19:16 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph Show raw failure time=2026-02-25T17:55:59.092Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-02-25T17:56:01.248Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra time=2026-02-25T17:56:02.425Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=41f16ef5151a15d838a02a56a0559f4405c126b36b8928aa7e037a1e9efa87dc time=2026-02-25T17:57:33.227Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8454912/providers/Microsoft.Resources/deployments/41f16ef5151a15d838a02a56a0559f4405c126b36b8928aa7e037a1e9efa87dc/operationStatuses/08584295647237856843\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8454912/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8454912/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j8454912\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Internal error\\\",\\r\\n \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report b86e066b-493f-4f0c-a6b1-b5f0bcb7d93f:2/25/2026 5:57:10 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
16:26:37	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head b1cdfeb · merge b1cdfeb	n/a	e2e	failed	★#4206 (merged)	1	e2e (1) Test Panicked 16:26:37 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure github.com/Azure/ARO-HCP/test/util/framework.Must[...](...) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:391 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000f51680?, {0x7f2a65373660?, 0xc0009f4120?}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:603 +0x37 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).NewResourceGroup(0xc000d9ed80, {0x7f2a65373660, 0xc0009f4120}, {0x7698547, 0x15}, {0xc000014109, 0x7}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5 github.com/Azure/ARO-HCP/test/e2e.init.func17.2({0x7f2a65373660, 0xc0009f4120}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:67 +0xe9 github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8076598?, 0xc0009f4120?}) /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e github.com/onsi/ginkgo/v2/internal.(Suite).runNode.func3() /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b created by github.com/onsi/ginkgo/v2/internal.(Suite).runNode in goroutine 79 /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50 fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:391]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:391]: Test Panicked github.com/Azure/ARO-HCP/test/util/framework.Must[...](...) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:391 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000f51680?, {0x7f2a65373660?, 0xc0009f4120?}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:603 +0x37 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).NewResourceGroup(0xc000d9ed80, {0x7f2a65373660, 0xc0009f4120}, {0x7698547, 0x15}, {0xc000014109, 0x7}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5 github.com/Azure/ARO-HCP/test/e2e.init.func17.2({0x7f2a65373660, 0xc0009f4120}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:67 +0xe9 github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8076598?, 0xc0009f4120?}) /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e github.com/onsi/ginkgo/v2/internal.(Suite).runNode.func3() /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b created by github.com/onsi/ginkgo/v2/internal.(Suite).runNode in goroutine 79 /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50 fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:391]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:391]: Test Panicked
16:18:41	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c8160d3	n/a	n/a	passed	#4208 (closed)	0	n/a
16:11:18	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 805cfcf	n/a	unknown	failed	#4033 (closed)	1	non-artifact-backed failure (no junit artifacts)
15:52:24	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 6724268 · merge 8700ec5	n/a	n/a	passed	#4119 (merged)	0	n/a
15:40:30	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 3ca804d · merge 3ca804d	n/a	provision	failed	★#3679 (merged)	1	provision (1) istio-1.28.3-linux-amd64.tar.gz: FAILED 15:40:30 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade · suite step graph Show raw failure time=2026-02-25T16:32:09.371Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n Kind: Shell\n Command: ./istio.sh\n" time=2026-02-25T16:32:18.368Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="error running Shell Step, failed to execute shell command: + set -euo pipefail\n+ ISTIO_NAMESPACE=aks-istio-system\n+ echo '******** Check istio is up and running **********'\n****** Check istio is up and running **********\n++ kubectl get pods -n aks-istio-system -l istio.io/rev=asm-1-26 --field-selector=status.phase=Running --no-headers\n++ wc -l\n+ ISTIO_PODS_COUNT=2\n+ [[ 2 -lt 2 ]]\n+ echo '****** Download istioctl **********'\n****** Download istioctl ************\n++ uname\n+ OS=Linux\n+ '[' Linux = Darwin ']'\n+ OSEXT=linux\n++ uname -m\n+ LOCAL_ARCH=x86_64\n+ case \"${LOCAL_ARCH}\" in\n+ ISTIO_ARCH=amd64\n+ ISTIO_URL=https://github.com/istio/istio/releases/download/1.28.3/istio-1.28.3-linux-amd64.tar.gz\n+ SHA256_URL=https://github.com/istio/istio/releases/download/1.28.3/istio-1.28.3-linux-amd64.tar.gz.sha256\n+ curl -sL https://github.com/istio/istio/releases/download/1.28.3/istio-1.28.3-linux-amd64.tar.gz -o istio-1.28.3-linux-amd64.tar.gz\n+ curl -sL https://github.com/istio/istio/releases/download/1.28.3/istio-1.28.3-linux-amd64.tar.gz.sha256 -o istio-1.28.3-linux-amd64.tar.gz.sha256\n+ sha256sum -c istio-1.28.3-linux-amd64.tar.gz.sha256\nistio-1.28.3-linux-amd64.tar.gz: FAILED\nsha256sum: WARNING: 1 computed checksum did NOT match\n exit status 1"
15:31:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 5702af3 · merge 5702af3	n/a	n/a	passed	#4167 (merged)	0	n/a
15:28:50	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 5702af3 · merge 5702af3	n/a	n/a	passed	#4167 (merged)	0	n/a
14:53:33	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head a02b279 · merge a02b279	n/a	n/a	passed	#4207 (merged)	0	n/a
14:45:18	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head d0eeb47 · merge e6b55da	n/a	n/a	passed	#4166 (merged)	0	n/a
14:34:51	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head a6ce4d3 · merge e5b4a5c	n/a	n/a	passed	#3764 (merged)	0	n/a
13:53:07	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 532d1af	n/a	n/a	passed	#4004 (open)	0	n/a
13:52:39	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head cbb5a98	n/a	n/a	passed	#4189 (open)	0	n/a
13:42:08	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2b12400	n/a	n/a	passed	#3913 (open)	0	n/a
13:38:55	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 8077c8c · merge e6b55da	n/a	unknown	failed	#4166 (merged)	1	non-artifact-backed failure (no junit artifacts)
13:24:45	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4ddffef · merge 4ddffef	n/a	n/a	passed	#4032 (merged)	0	n/a
13:23:14	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 197dda7 · merge e6b55da	n/a	unknown	failed	#4166 (merged)	1	non-artifact-backed failure (no junit artifacts)
13:05:52	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4bea39e	n/a	n/a	passed	#3094 (closed)	0	n/a
12:54:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 592efb2 · merge a36f37a	n/a	n/a	passed	#4205 (merged)	0	n/a
12:46:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 6a05d48 · merge 6a05d48	n/a	n/a	passed	#4201 (merged)	0	n/a
12:28:58	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 5702af3 · merge 5702af3	n/a	n/a	passed	#4167 (merged)	0	n/a
12:24:06	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 5704041 · merge 5704041	n/a	n/a	passed	#4204 (merged)	0	n/a
11:41:27	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head df979fb · merge df979fb	n/a	n/a	passed	#4200 (merged)	0	n/a
10:35:01	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 2714126 · merge 2714126	n/a	n/a	passed	#4056 (merged)	0	n/a
10:32:07	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 2714126 · merge 2714126	n/a	n/a	passed	#4056 (merged)	0	n/a
09:22:24	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head aa0d5ab · merge 4c15703	n/a	provision	failed	⚠#4075 (merged)	1	provision (1) resource not ready, name: aro-hcp-backend, kind: Deployment, status: Failed context deadline exceeded 09:22:24 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RP.Backend/service/deploy · suite step graph Show raw failure time=2026-02-25T10:13:11.423Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-02-25T10:13:12.107Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[azure:map[cloudEnvironmentName:AzurePublicCloud maestro:map[sourceEnvironmentIdentifier:arohcpprow] msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj8715648.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj8715648.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:d79c2524-9c8f-4718-b6cc-c7bbb6e7de0a databaseName:arohcpprow-rp-j8715648 databaseUrl:https://arohcpprow-rp-j8715648.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-0j40hnqt/pipeline@sha256:ce193edf2450d70f9b032f2015c940f702b34f49906b7373048f1451f8ac4daa replicas:2 zoneCount:4] fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-0j40hnqt/pipeline:pull workloadIdentityClientId:3d10ae00-294d-4cdb-b665-2f56cfc8dddb workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:d79c2524-9c8f-4718-b6cc-c7bbb6e7de0a workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-02-25T10:13:12.107Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-02-25T10:13:12.354Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp time=2026-02-25T10:13:12.354Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false time=2026-02-25T10:13:12.354Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-02-25T10:13:12.711Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-02-25T10:18:19.347Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" time=2026-02-25T10:18:19.347Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-02-25T10:18:19.347Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-02-25T10:18:22.143Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy release=aro-hcp-backend-dev namespace=aro-hcp status=failed description="Release \"aro-hcp-backend-dev\" failed: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[azure:map[cloudEnvironmentName:AzurePublicCloud maestro:map[sourceEnvironmentIdentifier:arohcpprow] msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj8715648.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj8715648.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:d79c2524-9c8f-4718-b6cc-c7bbb6e7de0a databaseName:arohcpprow-rp-j8715648 databaseUrl:https://arohcpprow-rp-j8715648.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-0j40hnqt/pipeline@sha256:ce193edf2450d70f9b032f2015c940f702b34f49906b7373048f1451f8ac4daa replicas:2 zoneCount:4] fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-0j40hnqt/pipeline:pull workloadIdentityClientId:3d10ae00-294d-4cdb-b665-2f56cfc8dddb workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:d79c2524-9c8f-4718-b6cc-c7bbb6e7de0a workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-02-25T10:18:22.145Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/4yTQY/aPhDF7/kUo5xAileQv/5VRbUHuu2p2gq1PbXqwbFfwYvjicYOiBUfvkpYSrQKLdfH7z3exDOZRyJB5FYMIt2T1UknXXlMuHqCSZ9csIuYxIV1QSfps67xR8IOoVdio81Znv7IiIjypZFV6/17F6wL67ygvNJmi2BV03qvqouuhdXGNHlxMq4AWbZpg5Cc0clxGEDqHFIjiTNxxP+FPTrZQ1uIgod5FXImP6DxfKgR0sg/jPBdKxb33JdasXfm0FPe8/5c6C/2rzCCtBLeOQt58DrG4WeJkJ0zUFsc1E63Po1G9MwjB5dYbmv94lkaw+1p0uvsA4dfbv2om2Ex/dwKlLQhuRrK9Mit3qt090oju/HvR3sZ58adyH6+y7ZtBQlIiB+7hY3ZkfYbCKibJyZdN1Qh7YFAE6sTOnmSl7PyjZqVqvz/23y2mP+3mJff8ynd3dFV5u2i7JhpdqQndoG2Lth7FwJkcGYc6HJdw7N6fU/ZkRrh7tdL0YIatif4tpCCPK8LqhGjXneRLBZC1WEwvEU0vwMAAP//MRvEEQwEAAA=" time=2026-02-25T10:18:22.147Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-02-25T10:18:22.147Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy step=deploy error="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: InProgress\ncontext deadline exceeded" time=2026-02-25T10:19:22.148Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-02-25T10:19:22.843Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[azure:map[cloudEnvironmentName:AzurePublicCloud maestro:map[sourceEnvironmentIdentifier:arohcpprow] msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj8715648.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj8715648.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:d79c2524-9c8f-4718-b6cc-c7bbb6e7de0a databaseName:arohcpprow-rp-j8715648 databaseUrl:https://arohcpprow-rp-j8715648.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-0j40hnqt/pipeline@sha256:ce193edf2450d70f9b032f2015c940f702b34f49906b7373048f1451f8ac4daa replicas:2 zoneCount:4] fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-0j40hnqt/pipeline:pull workloadIdentityClientId:3d10ae00-294d-4cdb-b665-2f56cfc8dddb workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:d79c2524-9c8f-4718-b6cc-c7bbb6e7de0a workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-02-25T10:19:22.843Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-02-25T10:19:23.133Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp time=2026-02-25T10:19:23.133Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false time=2026-02-25T10:19:23.133Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-02-25T10:19:23.308Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy numVersions=1 time=2026-02-25T10:24:29.268Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-backend, kind: Deployment, status: Failed\ncontext deadline exceeded" time=2026-02-25T10:24:29.268Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-02-25T10:24:29.268Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-02-25T10:24:33.348Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy release=aro-hcp-backend-dev namespace=aro-hcp status=failed description="Upgrade \"aro-hcp-backend-dev\" failed: resource not ready, name: aro-hcp-backend, kind: Deployment, status: Failed\ncontext deadline exceeded" values="map[azure:map[cloudEnvironmentName:AzurePublicCloud maestro:map[sourceEnvironmentIdentifier:arohcpprow] msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj8715648.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj8715648.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:d79c2524-9c8f-4718-b6cc-c7bbb6e7de0a databaseName:arohcpprow-rp-j8715648 databaseUrl:https://arohcpprow-rp-j8715648.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-0j40hnqt/pipeline@sha256:ce193edf2450d70f9b032f2015c940f702b34f49906b7373048f1451f8ac4daa replicas:2 zoneCount:4] fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-0j40hnqt/pipeline:pull workloadIdentityClientId:3d10ae00-294d-4cdb-b665-2f56cfc8dddb workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:d79c2524-9c8f-4718-b6cc-c7bbb6e7de0a workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-02-25T10:24:33.349Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/4yTQW8aMRCF7/srRnsK0joim7ZSqXJI056qVKjpqVUPXvsVHLye1dgLIuLHV4ZQVggUbqvnzzNv3noKj0SCyL0YRLojq5NOuvG44uYZJn1zwU5iEhdmFe2k77rFfwlLhK0SO2328uh3QURUPkGWzuDeGO5DKisqG20WCDZ/amE1N11Z7dgHDn/d7FF3A0zpl16gpA/JtVBmi1x69yz9gz2y7KEtRMHDJMfhBPnqf3Ci9tVbJHEmnrh0b2Tae//ZBevCbOio671XzUE/vtinOYt70dnNlL0z6y3lPa/2/dT5AJ9gBGkqvHQW8uB1jMPmcTeLWmCtlrr36UwyJ3y/HdQXdJ7XLUI6kdUJfgpIHhchOaOPil6Q8euPeeTgEssbPYs/n4pF30ACEuLX/GBjsaHVHALKDysm3XbUIK2AQFdWJ2T5qqzH9Qc1rlX9/ufNeHLzcVLf/ipHdH1N55j63eQ2M6NiQ8/sAi1csHcuBMhgzTjQYbuGa3W8T8WGOuF8ejBaUcd2B19WpCLPs4paxKhnuSSLhVCzHgxvEc2/AAAA//9WZwixDAQAAA==" time=2026-02-25T10:24:33.351Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-02-25T10:24:33.351Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy step=deploy error="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: Failed\ncontext deadline exceeded" time=2026-02-25T10:25:33.354Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-02-25T10:25:34.105Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[azure:map[cloudEnvironmentName:AzurePublicCloud maestro:map[sourceEnvironmentIdentifier:arohcpprow] msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj8715648.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj8715648.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:d79c2524-9c8f-4718-b6cc-c7bbb6e7de0a databaseName:arohcpprow-rp-j8715648 databaseUrl:https://arohcpprow-rp-j8715648.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-0j40hnqt/pipeline@sha256:ce193edf2450d70f9b032f2015c940f702b34f49906b7373048f1451f8ac4daa replicas:2 zoneCount:4] fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-0j40hnqt/pipeline:pull workloadIdentityClientId:3d10ae00-294d-4cdb-b665-2f56cfc8dddb workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:d79c2524-9c8f-4718-b6cc-c7bbb6e7de0a workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-02-25T10:25:34.105Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-02-25T10:25:34.467Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp time=2026-02-25T10:25:34.467Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false time=2026-02-25T10:25:34.467Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-02-25T10:25:34.691Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy numVersions=2 time=2026-02-25T10:30:40.986Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource not ready, name: aro-hcp-backend, kind: Deployment, status: Failed\ncontext deadline exceeded" time=2026-02-25T10:30:40.986Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-02-25T10:30:40.986Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-02-25T10:30:45.535Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy release=aro-hcp-backend-dev namespace=aro-hcp status=failed description="Upgrade \"aro-hcp-backend-dev\" failed: resource not ready, name: aro-hcp-backend, kind: Deployment, status: Failed\ncontext deadline exceeded" values="map[azure:map[cloudEnvironmentName:AzurePublicCloud maestro:map[sourceEnvironmentIdentifier:arohcpprow] msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj8715648.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj8715648.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:d79c2524-9c8f-4718-b6cc-c7bbb6e7de0a databaseName:arohcpprow-rp-j8715648 databaseUrl:https://arohcpprow-rp-j8715648.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-0j40hnqt/pipeline@sha256:ce193edf2450d70f9b032f2015c940f702b34f49906b7373048f1451f8ac4daa replicas:2 zoneCount:4] fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-0j40hnqt/pipeline:pull workloadIdentityClientId:3d10ae00-294d-4cdb-b665-2f56cfc8dddb workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:d79c2524-9c8f-4718-b6cc-c7bbb6e7de0a workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-02-25T10:30:45.536Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/4yTMW/bMBCFd/2Kg6YYEAPXiTu4yOCmnYoURtMpRQeKfJUZU6RwPNlw4B9fyI5rwVAab8Lj946nd7zMQ4iRYssGie7IatGiS4+rWD7DyDcX7CwJu1AVdJC+6xr/JKwR9kpqtDnKo18ZEVH+CF47g7kxsQ2SF5SX2qwQbPepOaqlafLiwP6IHp9dsC5UPVB5aAtW8DDiYhgwvl7SO1FHcw1hZ9KAaQHwvJUlgjijzypf4H+EYciC49pZ8L3XKfW7Toee1Apbtdatl4ES9zH8cdWDbvpG/dIyFLdBXA1l9sil3jfpLtpOfj/LL2h83NYIMhDHAD83vGi9H5hb03qvypN+bmxlGdm97KNfRO/Mdk95HzfH2P9z7+vMH2JwEvmdXrPfn7JVW4IDBOlr92BTtqPNEgzqck6i64ZKyAYIdGW1oJOv8sl48lGNJ2oy/flhPJtMZze3T/mIrq/pLeZmPLudPuWjUbaj5+gCrVywdy4EcG/NYqDTdvXX6nyfsh01HLvTU6MFNdEe4MuKFORjVVCNlHTVlYxswVRuez9vkczfAAAA//8AiQ0nDAQAAA==" time=2026-02-25T10:30:45.538Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-02-25T10:30:45.538Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy step=deploy error="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: Failed\ncontext deadline exceeded" time=2026-02-25T10:31:45.544Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: aro-hcp-backend, kind: Deployment, status: Failed\ncontext deadline exceeded"
09:02:12	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 9ed1eef · merge e5b4a5c	n/a	n/a	passed	#3764 (merged)	0	n/a
09:01:30	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 84c6ad3 · merge 3ca804d	n/a	n/a	passed	#3679 (merged)	0	n/a
08:48:10	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head a13199e · merge 4c15703	n/a	unknown	failed	#4075 (merged)	1	non-artifact-backed failure (no junit artifacts)
08:34:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 6e62bfe · merge 6e62bfe	n/a	n/a	passed	#4203 (merged)	0	n/a
08:26:55	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 97f1689 · merge 97f1689	n/a	n/a	passed	#4202 (merged)	0	n/a
08:11:03	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head dc5fc75 · merge b541ca7	n/a	n/a	passed	#4196 (merged)	0	n/a
07:53:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 1b7a82b · merge 1b7a82b	n/a	n/a	passed	#4197 (merged)	0	n/a
07:42:54	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 2714126 · merge 2714126	n/a	n/a	passed	#4056 (merged)	0	n/a
07:42:16	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 6a05d48 · merge 6a05d48	n/a	n/a	passed	#4201 (merged)	0	n/a
07:36:49	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head e5ce62a · merge 3ca804d	n/a	provision	failed	⚠#3679 (merged)	1	provision (1) Error response from registry: generic storage error: unknown_storage_failure 07:36:49 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SessionGate/global/mirror-image · suite step graph Show raw failure time=2026-02-25T08:27:10.553Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image description="Step mirror-image\n Kind: ImageMirror\n From {<nil> sessiongate.image.registry <nil>}:{<nil> sessiongate.image.repository <nil>}@{<nil> sessiongate.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T08:27:38.234Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nSetting up registry authentication for CI source registry.\ninfo: Using registry public hostname registry.build06.ci.openshift.org\nSaved credentials for registry.build06.ci.openshift.org into /tmp/tmp.ZkrpcQnKXQ/containers/auth.json\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.build06.ci.openshift.org/ci-op-6z71dbgb/pipeline@sha256:571a3a553a4b3fdb3bf4625d9a5f67b111107b53b5ca9c402ec663d2ba40a5ce to arohcpsvcdev.azurecr.io/ci-op-6z71dbgb/pipeline:571a3a553a4b3fdb3bf4625d9a5f67b111107b53b5ca9c402ec663d2ba40a5ce.\nThe image will still be available under it's original digest sha256:571a3a553a4b3fdb3bf4625d9a5f67b111107b53b5ca9c402ec663d2ba40a5ce in the target registry.\nError response from registry: generic storage error: unknown_storage_failure\n exit status 1"
07:30:53	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head df979fb · merge df979fb	n/a	unknown	failed	#4200 (merged)	1	non-artifact-backed failure (no junit artifacts)
05:50:13	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head f357ca7 · merge f357ca7	n/a	provision	failed	★#4199 (merged)	1	provision (1) Inner error: { 05:50:13 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RP.HypershiftOperator/global/mirror-image · suite step graph Show raw failure time=2026-02-25T06:36:55.477Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.HypershiftOperator resourceGroup=global step=mirror-image description="Step mirror-image\n Kind: ImageMirror\n From {<nil> hypershift.image.registry <nil>}:{<nil> hypershift.image.repository <nil>}@{<nil> hypershift.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T06:36:57.027Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.HypershiftOperator resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nERROR: (Forbidden) Operation get is not allowed on a disabled secret.\nCode: Forbidden\nMessage: Operation get is not allowed on a disabled secret.\nInner error: {\n \"code\": \"SecretDisabled\"\n}\n exit status 1"
05:02:46	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 9ff64c7 · merge ad7003e	n/a	provision	failed	★#4188 (merged)	1	provision (1) Inner error: { 05:02:46 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-azure-plugin-image · suite step graph Show raw failure time=2026-02-25T05:21:51.649Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image description="Step mirror-azure-plugin-image\n Kind: ImageMirror\n From {<nil> velero.azurePlugin.registry <nil>}:{<nil> velero.azurePlugin.repository <nil>}@{<nil> velero.azurePlugin.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T05:21:53.022Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nERROR: (Forbidden) Operation get is not allowed on a disabled secret.\nCode: Forbidden\nMessage: Operation get is not allowed on a disabled secret.\nInner error: {\n \"code\": \"SecretDisabled\"\n}\n exit status 1"
03:02:45	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 9ff64c7 · merge ad7003e	n/a	provision	failed	★#4188 (merged)	1	provision (1) ERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4 03:02:45 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SecretSyncController/global/mirror-image · suite step graph Show raw failure time=2026-02-25T03:54:32.710Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image description="Step mirror-image\n Kind: ImageMirror\n From {<nil> secretSyncController.image.registry <nil>}:{<nil> secretSyncController.image.repository <nil>}@{<nil> secretSyncController.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T03:54:34.346Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.k8s.io from arohcpdev-global KV.\nERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4\n exit status 1"
02:10:44	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 9fdaef4 · merge 9fdaef4	n/a	provision	failed	★#4134 (merged)	1	provision (1) ERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4 02:10:44 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-velero-image · suite step graph Show raw failure time=2026-02-25T03:03:01.221Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image description="Step mirror-velero-image\n Kind: ImageMirror\n From {<nil> velero.server.registry <nil>}:{<nil> velero.server.repository <nil>}@{<nil> velero.server.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T03:03:02.633Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4\n exit status 1"
01:27:09	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 5702af3 · merge 5702af3	n/a	provision	failed	★#4167 (merged)	1	provision (1) ERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4 01:27:09 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Maestro.Server/global/mirror-image · suite step graph Show raw failure time=2026-02-25T02:17:47.824Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=global step=mirror-image description="Step mirror-image\n Kind: ImageMirror\n From {<nil> maestro.image.registry <nil>}:{<nil> maestro.image.repository <nil>}@{<nil> maestro.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T02:17:49.114Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4\n exit status 1"
01:24:30	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 5702af3 · merge 5702af3	n/a	provision	failed	★#4167 (merged)	1	provision (1) ERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4 01:24:30 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Maestro.Agent/global/mirror-image · suite step graph Show raw failure time=2026-02-25T02:13:06.734Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=global step=mirror-image description="Step mirror-image\n Kind: ImageMirror\n From {<nil> maestro.image.registry <nil>}:{<nil> maestro.image.repository <nil>}@{<nil> maestro.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T02:13:08.260Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4\n exit status 1"
01:21:40	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 600af20 · merge 3ca804d	n/a	provision	failed	★#3679 (merged)	1	provision (1) ERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4 01:21:40 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Maestro.Server/global/mirror-image · suite step graph Show raw failure time=2026-02-25T02:12:16.259Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=global step=mirror-image description="Step mirror-image\n Kind: ImageMirror\n From {<nil> maestro.image.registry <nil>}:{<nil> maestro.image.repository <nil>}@{<nil> maestro.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T02:12:17.962Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4\n exit status 1"
01:13:32	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 600af20 · merge 3ca804d	n/a	provision	failed	★#3679 (merged)	1	provision (1) ERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4 01:13:32 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Maestro.Agent/global/mirror-image · suite step graph Show raw failure time=2026-02-25T01:56:44.376Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=global step=mirror-image description="Step mirror-image\n Kind: ImageMirror\n From {<nil> maestro.image.registry <nil>}:{<nil> maestro.image.repository <nil>}@{<nil> maestro.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T01:56:46.584Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4\n exit status 1"
00:32:29	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0c69fe5	n/a	provision	failed	★#4169 (closed)	1	provision (1) ERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4 00:32:29 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-velero-image · suite step graph Show raw failure time=2026-02-25T01:19:21.853Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image description="Step mirror-velero-image\n Kind: ImageMirror\n From {<nil> velero.server.registry <nil>}:{<nil> velero.server.repository <nil>}@{<nil> velero.server.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T01:19:23.238Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4\n exit status 1"
00:03:51	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 9fdaef4 · merge 9fdaef4	n/a	provision	failed	★#4134 (merged)	1	provision (1) ERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4 00:03:51 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SecretSyncController/global/mirror-image · suite step graph Show raw failure time=2026-02-25T00:54:51.286Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image description="Step mirror-image\n Kind: ImageMirror\n From {<nil> secretSyncController.image.registry <nil>}:{<nil> secretSyncController.image.repository <nil>}@{<nil> secretSyncController.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T00:54:52.638Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry registry.k8s.io from arohcpdev-global KV.\nERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4\n exit status 1"
00:01:06	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 9fdaef4 · merge 9fdaef4	n/a	provision	failed	★#4134 (merged)	1	provision (1) ERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4 00:01:06 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Maestro.Agent/global/mirror-image · suite step graph Show raw failure time=2026-02-25T00:59:54.329Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=global step=mirror-image description="Step mirror-image\n Kind: ImageMirror\n From {<nil> maestro.image.registry <nil>}:{<nil> maestro.image.repository <nil>}@{<nil> maestro.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-25T00:59:56.175Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nERROR: Invalid base64-encoded string: number of data characters (1325) cannot be 1 more than a multiple of 4\n exit status 1"