CIHealth Overview

Executive Status

Env	Goal	Runs	Success	Gap vs target	Change vs prev	Provision success	Provision change vs prev	E2E success	E2E success vs prev
DEV	95% - Post-good runs + Tide batches	136	50.00% (off track)	-45.00pp	-13.89pp	87.29% (103/118)	-5.79pp	66.02% (68/103)	-10.01pp
INT	95% - All E2E job runs	24	70.83% (off track)	-24.17pp	+23.21pp	n/a	n/a	70.83% (17/24)	+23.21pp
STG	95% - All E2E job runs	20	25.00% (off track)	-70.00pp	-17.11pp	n/a	n/a	25.00% (5/20)	-19.44pp
PROD	95% - All E2E job runs	34	32.35% (off track)	-62.65pp	-21.14pp	n/a	n/a	32.35% (11/34)	-23.74pp

Environment: DEV

E2E Jobs

333

Success Rate

36.34%

Provision success

76.98% (214/278)

E2E success

56.54% (121/214)

E2E Jobs (post-good + batches)

136

Success Rate (post-good + batches)

50.00%

Provision success (post-good + batches)

87.29% (103/118)

E2E success (post-good + batches)

66.02% (68/103)

Daily Run Outcomes

Successful runs Provision failures E2E failures Other failures

64 runs

75 runs

98 runs

69 runs

22 runs

3 runs

2 runs

Daily Run Outcomes for DEV Goal Basis (post-good + batches)

Successful runs (post-good + batches) Provision failures E2E failures Other failures

18 runs

25 runs

47 runs

41 runs

5 runs

0 runs

No runs

2026-04-05 UTC

0 runs

No runs

Pass rate	Runs	Date (UTC)	Suite	Test
84.34%	198	2026-04-07	rp-api-compat-all/parallel	Customer should be able to create a cluster with an external auth config and get the external auth config
86.87%	198	2026-04-07	rp-api-compat-all/parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP
89.90%	198	2026-04-07	rp-api-compat-all/parallel	Customer should be able to update nodepool replicas and autoscaling
92.25%	142	2026-04-07	rp-api-compat-all/parallel	Customer should be able to create an HCP cluster
93.94%	198	2026-04-07	rp-api-compat-all/parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully

Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.

Jump to the full Failure Patterns view for this window

Failure pattern

Failed at

Trend

Also in

add password: error status code received from the API

full failure pattern:

add password: error status code received from the API

Signal:

e2e

INT, PROD, STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:130]: Unexpected error:
    <*fmt.wrapError | 0xc000a69040>: 
    add password: error status code received from the API
    {
        msg: "add password: error status code received from the API",
        err: <*odataerrors.ODataError | 0xc000732270>{
            ApiError: {
                Message: "",
                ResponseStatusCode: 404,
                ResponseHeaders: {
                    header: {
                        headers: {
                            "request-id": {
                                "d5af5132-06b4-4d00-bd50-155c478afb64": {},
                            },
                            "cache-control": {"no-cache": {}},
                            "x-ms-ags-diagnostic": {
                                "{\"ServerInfo\":{\"DataCenter\":\"Canada Central\",\"Slice\":\"E\",\"Ring\":\"3\",\"ScaleUnit\":\"001\",\"RoleInstance\":\"YT1PEPF000041CC\"}}": {},
                            },
                            "client-request-id": {
                                "d5af5132-06b4-4d00-bd50-155c478afb64": {},
                            },
                            "x-ms-resource-unit": {"1": {}},
                            "date": {
                                "Wed, 01 Apr 2026 18:38:02 GMT": {},
                            },
                            "vary": {
                                "Accept-Encoding": {},
                            },
                            "strict-transport-security": {
                                "max-age=31536000": {},
                            },
                            "content-type": {
                                "application/json": {},
                            },
                        },
                    },
                },
            },
            backingStore: <*store.InMemoryBackingStore | 0xc000a68dc0>{
                returnOnlyChangedValues: false,
                initializationCompleted: false,
                store: {
                    "errorEscaped": <*odataerrors.MainError | 0xc000aea2b0>{
                        backingStore: <*store.InMemoryBackingStore | 0xc000a68ee0>{
                            returnOnlyChangedValues: false,
                            initializationCompleted: false,
                            store: {
                                "code": <*string | 0xc000aea030>"Request_ResourceNotFound",
                                "message": <*string | 0xc000aea0a0>"Resource 'f3be3f65-4e9b-4a1d-b628-2b6e7f7bbc19' does not exist or one of its queried reference-property objects are not present.",
                                "innerError": <*odataerrors.InnerError | 0xc000aea310>{
                                    backingStore: <*store.InMemoryBackingStore | 0xc000a68f80>{
                                        returnOnlyChangedValues: false,
                                        initializationCompleted: false,
                                        store: {...: ..., ...: ...},
                                        subscribers: {},
                                        changedValues: {...: ..., ...: ...},
                                    },
                                },
                            },
                            subscribers: {},
                            changedValues: {"message": false, "innerError": false, "code": false},
                        },
                    },
                },
                subscribers: {},
                changedValues: {
                    "errorEscaped": false,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:130]: Unexpected error:
    <*fmt.wrapError | 0xc000a69040>: 
    add password: error status code received from the API
    {
        msg: "add password: error status code received from the API",
        err: <*odataerrors.ODataError | 0xc000732270>{
            ApiError: {
                Message: "",
                ResponseStatusCode: 404,
                ResponseHeaders: {
                    header: {
                        headers: {
                            "request-id": {
                                "d5af5132-06b4-4d00-bd50-155c478afb64": {},
                            },
                            "cache-control": {"no-cache": {}},
                            "x-ms-ags-diagnostic": {
                                "{\"ServerInfo\":{\"DataCenter\":\"Canada Central\",\"Slice\":\"E\",\"Ring\":\"3\",\"ScaleUnit\":\"001\",\"RoleInstance\":\"YT1PEPF000041CC\"}}": {},
                            },
                            "client-request-id": {
                                "d5af5132-06b4-4d00-bd50-155c478afb64": {},
                            },
                            "x-ms-resource-unit": {"1": {}},
                            "date": {
                                "Wed, 01 Apr 2026 18:38:02 GMT": {},
                            },
                            "vary": {
                                "Accept-Encoding": {},
                            },
                            "strict-transport-security": {
                                "max-age=31536000": {},
                            },
                            "content-type": {
                                "application/json": {},
                            },
                        },
                    },
                },
            },
            backingStore: <*store.InMemoryBackingStore | 0xc000a68dc0>{
                returnOnlyChangedValues: false,
                initializationCompleted: false,
                store: {
                    "errorEscaped": <*odataerrors.MainError | 0xc000aea2b0>{
                        backingStore: <*store.InMemoryBackingStore | 0xc000a68ee0>{
                            returnOnlyChangedValues: false,
                            initializationCompleted: false,
                            store: {
                                "code": <*string | 0xc000aea030>"Request_ResourceNotFound",
                                "message": <*string | 0xc000aea0a0>"Resource 'f3be3f65-4e9b-4a1d-b628-2b6e7f7bbc19' does not exist or one of its queried reference-property objects are not present.",
                                "innerError": <*odataerrors.InnerError | 0xc000aea310>{
                                    backingStore: <*store.InMemoryBackingStore | 0xc000a68f80>{
                                        returnOnlyChangedValues: false,
                                        initializationCompleted: false,
                                        store: {...: ..., ...: ...},
                                        subscribers: {},
                                        changedValues: {...: ..., ...: ...},
                                    },
                                },
                            },
                            subscribers: {},
                            changedValues: {"message": false, "innerError": false, "code": false},
                        },
                    },
                },
                subscribers: {},
                changedValues: {
                    "errorEscaped": false,
                },
            },
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:130]: Unexpected error:
    <*fmt.wrapError | 0xc00075d220>: 
    add password: error status code received from the API
    {
        msg: "add password: error status code received from the API",
        err: <*odataerrors.ODataError | 0xc0015804e0>{
            ApiError: {
                Message: "",
                ResponseStatusCode: 404,
                ResponseHeaders: {
                    header: {
                        headers: {
                            "content-type": {
                                "application/json": {},
                            },
                            "strict-transport-security": {
                                "max-age=31536000": {},
                            },
                            "date": {
                                "Wed, 01 Apr 2026 13:08:29 GMT": {},
                            },
                            "cache-control": {"no-cache": {}},
                            "request-id": {
                                "35e515d8-221d-4305-a61f-d52ca4501395": {},
                            },
                            "x-ms-resource-unit": {"1": {}},
                            "client-request-id": {
                                "35e515d8-221d-4305-a61f-d52ca4501395": {},
                            },
                            "x-ms-ags-diagnostic": {
                                "{\"ServerInfo\":{\"DataCenter\":\"Canada Central\",\"Slice\":\"E\",\"Ring\":\"3\",\"ScaleUnit\":\"001\",\"RoleInstance\":\"YT1PEPF000041CD\"}}": {},
                            },
                            "vary": {
                                "Accept-Encoding": {},
                            },
                        },
                    },
                },
            },
            backingStore: <*store.InMemoryBackingStore | 0xc00075d1c0>{
                returnOnlyChangedValues: false,
                initializationCompleted: false,
                store: {
                    "errorEscaped": <*odataerrors.MainError | 0xc0008094d0>{
                        backingStore: <*store.InMemoryBackingStore | 0xc00075d1e0>{
                            returnOnlyChangedValues: false,
                            initializationCompleted: false,
                            store: {
                                "code": <*string | 0xc000809350>"Request_ResourceNotFound",
                                "message": <*string | 0xc0008093a0>"Resource '69be2020-1a4c-469e-b0f4-e62c871a9ee5' does not exist or one of its queried reference-property objects are not present.",
                                "innerError": <*odataerrors.InnerError | 0xc000809530>{
                                    backingStore: <*store.InMemoryBackingStore | 0xc00075d200>{
                                        returnOnlyChangedValues: false,
                                        initializationCompleted: false,
                                        store: {...: ..., ...: ...},
                                        subscribers: {},
                                        changedValues: {...: ..., ...: ...},
                                    },
                                },
                            },
                            subscribers: {},
                            changedValues: {"code": false, "message": false, "innerError": false},
                        },
                    },
                },
                subscribers: {},
                changedValues: {
                    "errorEscaped": false,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:130]: Unexpected error:
    <*fmt.wrapError | 0xc00075d220>: 
    add password: error status code received from the API
    {
        msg: "add password: error status code received from the API",
        err: <*odataerrors.ODataError | 0xc0015804e0>{
            ApiError: {
                Message: "",
                ResponseStatusCode: 404,
                ResponseHeaders: {
                    header: {
                        headers: {
                            "content-type": {
                                "application/json": {},
                            },
                            "strict-transport-security": {
                                "max-age=31536000": {},
                            },
                            "date": {
                                "Wed, 01 Apr 2026 13:08:29 GMT": {},
                            },
                            "cache-control": {"no-cache": {}},
                            "request-id": {
                                "35e515d8-221d-4305-a61f-d52ca4501395": {},
                            },
                            "x-ms-resource-unit": {"1": {}},
                            "client-request-id": {
                                "35e515d8-221d-4305-a61f-d52ca4501395": {},
                            },
                            "x-ms-ags-diagnostic": {
                                "{\"ServerInfo\":{\"DataCenter\":\"Canada Central\",\"Slice\":\"E\",\"Ring\":\"3\",\"ScaleUnit\":\"001\",\"RoleInstance\":\"YT1PEPF000041CD\"}}": {},
                            },
                            "vary": {
                                "Accept-Encoding": {},
                            },
                        },
                    },
                },
            },
            backingStore: <*store.InMemoryBackingStore | 0xc00075d1c0>{
                returnOnlyChangedValues: false,
                initializationCompleted: false,
                store: {
                    "errorEscaped": <*odataerrors.MainError | 0xc0008094d0>{
                        backingStore: <*store.InMemoryBackingStore | 0xc00075d1e0>{
                            returnOnlyChangedValues: false,
                            initializationCompleted: false,
                            store: {
                                "code": <*string | 0xc000809350>"Request_ResourceNotFound",
                                "message": <*string | 0xc0008093a0>"Resource '69be2020-1a4c-469e-b0f4-e62c871a9ee5' does not exist or one of its queried reference-property objects are not present.",
                                "innerError": <*odataerrors.InnerError | 0xc000809530>{
                                    backingStore: <*store.InMemoryBackingStore | 0xc00075d200>{
                                        returnOnlyChangedValues: false,
                                        initializationCompleted: false,
                                        store: {...: ..., ...: ...},
                                        subscribers: {},
                                        changedValues: {...: ..., ...: ...},
                                    },
                                },
                            },
                            subscribers: {},
                            changedValues: {"code": false, "message": false, "innerError": false},
                        },
                    },
                },
                subscribers: {},
                changedValues: {
                    "errorEscaped": false,
                },
            },
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:130]: Unexpected error:
    <*fmt.wrapError | 0xc00146d2e0>: 
    add password: error status code received from the API
    {
        msg: "add password: error status code received from the API",
        err: <*odataerrors.ODataError | 0xc0009e8360>{
            ApiError: {
                Message: "",
                ResponseStatusCode: 404,
                ResponseHeaders: {
                    header: {
                        headers: {
                            "cache-control": {"no-cache": {}},
                            "x-ms-ags-diagnostic": {
                                "{\"ServerInfo\":{\"DataCenter\":\"East US\",\"Slice\":\"E\",\"Ring\":\"5\",\"ScaleUnit\":\"000\",\"RoleInstance\":\"BL6PEPF0005C51F\"}}": {},
                            },
                            "date": {
                                "Wed, 01 Apr 2026 13:10:53 GMT": {},
                            },
                            "vary": {
                                "Accept-Encoding": {},
                            },
                            "x-ms-resource-unit": {"1": {}},
                            "content-type": {
                                "application/json": {},
                            },
                            "client-request-id": {
                                "27615470-0f8d-4dde-b750-bd09f093d147": {},
                            },
                            "strict-transport-security": {
                                "max-age=31536000": {},
                            },
                            "request-id": {
                                "27615470-0f8d-4dde-b750-bd09f093d147": {},
                            },
                        },
                    },
                },
            },
            backingStore: <*store.InMemoryBackingStore | 0xc00146d280>{
                returnOnlyChangedValues: false,
                initializationCompleted: false,
                store: {
                    "errorEscaped": <*odataerrors.MainError | 0xc0009c6180>{
                        backingStore: <*store.InMemoryBackingStore | 0xc00146d2a0>{
                            returnOnlyChangedValues: false,
                            initializationCompleted: false,
                            store: {
                                "message": <*string | 0xc001463f30>"Resource 'e4ef282d-f70e-4855-8109-37e8198a9256' does not exist or one of its queried reference-property objects are not present.",
                                "innerError": <*odataerrors.InnerError | 0xc0009c6210>{
                                    backingStore: <*store.InMemoryBackingStore | 0xc00146d2c0>{
                                        returnOnlyChangedValues: false,
                                        initializationCompleted: false,
                                        store: {...: ..., ...: ...},
                                        subscribers: {},
                                        changedValues: {...: ..., ...: ...},
                                    },
                                },
                                "code": <*string | 0xc001463ee0>"Request_ResourceNotFound",
                            },
                            subscribers: {},
                            changedValues: {"message": false, "innerError": false, "code": false},
                        },
                    },
                },
                subscribers: {},
                changedValues: {
                    "errorEscaped": false,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:130]: Unexpected error:
    <*fmt.wrapError | 0xc00146d2e0>: 
    add password: error status code received from the API
    {
        msg: "add password: error status code received from the API",
        err: <*odataerrors.ODataError | 0xc0009e8360>{
            ApiError: {
                Message: "",
                ResponseStatusCode: 404,
                ResponseHeaders: {
                    header: {
                        headers: {
                            "cache-control": {"no-cache": {}},
                            "x-ms-ags-diagnostic": {
                                "{\"ServerInfo\":{\"DataCenter\":\"East US\",\"Slice\":\"E\",\"Ring\":\"5\",\"ScaleUnit\":\"000\",\"RoleInstance\":\"BL6PEPF0005C51F\"}}": {},
                            },
                            "date": {
                                "Wed, 01 Apr 2026 13:10:53 GMT": {},
                            },
                            "vary": {
                                "Accept-Encoding": {},
                            },
                            "x-ms-resource-unit": {"1": {}},
                            "content-type": {
                                "application/json": {},
                            },
                            "client-request-id": {
                                "27615470-0f8d-4dde-b750-bd09f093d147": {},
                            },
                            "strict-transport-security": {
                                "max-age=31536000": {},
                            },
                            "request-id": {
                                "27615470-0f8d-4dde-b750-bd09f093d147": {},
                            },
                        },
                    },
                },
            },
            backingStore: <*store.InMemoryBackingStore | 0xc00146d280>{
                returnOnlyChangedValues: false,
                initializationCompleted: false,
                store: {
                    "errorEscaped": <*odataerrors.MainError | 0xc0009c6180>{
                        backingStore: <*store.InMemoryBackingStore | 0xc00146d2a0>{
                            returnOnlyChangedValues: false,
                            initializationCompleted: false,
                            store: {
                                "message": <*string | 0xc001463f30>"Resource 'e4ef282d-f70e-4855-8109-37e8198a9256' does not exist or one of its queried reference-property objects are not present.",
                                "innerError": <*odataerrors.InnerError | 0xc0009c6210>{
                                    backingStore: <*store.InMemoryBackingStore | 0xc00146d2c0>{
                                        returnOnlyChangedValues: false,
                                        initializationCompleted: false,
                                        store: {...: ..., ...: ...},
                                        subscribers: {},
                                        changedValues: {...: ..., ...: ...},
                                    },
                                },
                                "code": <*string | 0xc001463ee0>"Request_ResourceNotFound",
                            },
                            subscribers: {},
                            changedValues: {"message": false, "innerError": false, "code": false},
                        },
                    },
                },
                subscribers: {},
                changedValues: {
                    "errorEscaped": false,
                },
            },
        },
    }
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	22
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	11

Affected runs (31)

Date (UTC)	Associated PR	Prow job
2026-04-01 17:37	PR #4723	2039396816910290944
2026-04-01 12:02	PR #4685	2039312432748302336
2026-04-01 12:00	PR #4718	2039311951376420864
2026-04-01 11:10	PR #4705	2039299315725242368
2026-04-01 11:02	PR #4479	2039297255281790976
2026-04-01 09:28	PR #4673	2039273635952201728
2026-04-01 09:26	PR #4696	2039273255478497280
2026-04-01 08:44	PR #4599	2039262573844500480
2026-04-01 04:23	PR #4682	2039197037638979584
2026-04-01 03:27	PR #4564	2039182944043536384
2026-04-01 02:17	PR #4549	2039165221246865408
2026-03-31 22:28	PR #4564	2039107641015275520
2026-03-31 20:35	PR #4712	2039079132695891968
2026-03-31 20:10	PR #4133	2039072786458087424
2026-03-31 20:09	PR #4198	2039072526239272960
2026-03-31 19:26	PR #4703	2039061900968333312
2026-03-31 18:55	PR #4691	2039053973956071424
2026-03-31 16:54	PR #4705	2039023528509968384
2026-03-31 16:43	PR #4652	2039020878435454976
2026-03-31 16:08	PR #4610	2039012041993228288
2026-03-31 16:07	PR #4689	2039011689407451136
2026-03-31 15:42	PR #4693	2039005351646334976
2026-03-31 15:19	PR #4702	2038999740795850752
2026-03-31 14:21	PR #4210	2038985016473554944
2026-03-31 14:21	PR #4599	2038984939126394880
2026-03-31 14:04	PR #4566	2038980865790316544
2026-03-31 11:21	PR #4702	2038939677238497280
2026-03-31 10:57	PR #4658	2038933682185572352
2026-03-31 10:46	PR #4704	2038930809531928576
2026-03-31 07:45	PR #4698	2038885453414797312
2026-03-31 07:11	PR #4364	2038876767229317120

UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded

full failure pattern:

UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009657a0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-vdlznq" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vdlznq, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-vdlznq\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vdlznq, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0007d3b30>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vdlznq",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009657a0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-vdlznq" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vdlznq, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-vdlznq\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vdlznq, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0007d3b30>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-vdlznq",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a67860>: 
    failed waiting for nodepool="np-update-nodes" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-46csdr" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-46csdr, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-update-nodes\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-46csdr\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-46csdr, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000995eb0>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-46csdr",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a67860>: 
    failed waiting for nodepool="np-update-nodes" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-46csdr" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-46csdr, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-update-nodes\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-46csdr\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-46csdr, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000995eb0>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-nodes in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-46csdr",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:206]: Unexpected error:
    <*fmt.wrapErrors | 0xc00090ba70>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-hs6rh7" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hs6rh7, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-hs6rh7\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hs6rh7, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010b9110>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hs6rh7",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:206]: Unexpected error:
    <*fmt.wrapErrors | 0xc00090ba70>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-hs6rh7" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hs6rh7, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-hs6rh7\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hs6rh7, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010b9110>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hs6rh7",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

Contributing tests (2)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	17
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	8

Affected runs (22)

Date (UTC)	Associated PR	Prow job
2026-04-02 16:36	PR #4720	2039743836728594432
2026-04-02 16:32	PR #4688	2039742819844755456
2026-04-02 16:30	PR #4687	2039742386447323136
2026-04-02 13:57	PR #4742	2039703704566763520
2026-04-02 13:56	PR #4210	2039703492766994432
2026-04-02 13:47	PR #4700	2039701224894238720
2026-04-02 06:19	PR #4723	2039588637628174336
2026-04-02 06:07	PR #4702	2039585436556333056
2026-04-02 05:57	PR #4630	2039583088895332352
2026-04-01 19:11	PR #4724	2039420430925697024
2026-04-01 11:43	PR #4716	2039307567552794624
2026-04-01 11:02	PR #4479	2039297255281790976
2026-04-01 10:53	PR #4507	2039295185187246080
2026-04-01 09:29	PR #4664	2039274042879381504
2026-04-01 09:25	PR #4708	2039273039144685568
2026-04-01 08:52	PR #4626	2039264582718984192
2026-04-01 08:44	PR #4599	2039262573844500480
2026-03-31 16:47	PR #4711	2039021820367081472
2026-03-31 14:16	PR #4707	2038983708219805696
2026-03-31 13:41	PR #4705	2038975010877476864
2026-03-31 10:46	PR #4704	2038930809531928576
2026-03-31 10:27	PR #4689	2038926175870390272

ERROR CODE: DeploymentFailed; detail code OperationNotAllowed; detail message Operation could not be completed as it results in exceeding approved StandardDdsv6Family Cores quota....

full failure pattern:

ERROR CODE: DeploymentFailed; detail code OperationNotAllowed; detail message Operation could not be completed as it results in exceeding approved StandardDdsv6Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: WestUS3, Current Limit: 350, Current Usage: 344, Additional Required: 8, (Minimum) New Limit Required: 352. Setup Alerts when Quota reaches threshold. Learn more at <url> . Submit a request for Quota increase at <url> by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at <url>; provider Microsoft.Compute

Signal:

provision

none

Full failure examples (3)

time=2026-03-31T10:19:09.008Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-31T10:19:10.641Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-31T10:19:12.834Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=0e2d10aecc9dee2ed3db4bdb9b7ef1c04b366c5a329e9fc125a96e2435247e25
time=2026-03-31T10:33:46.818Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j5930368-svc/providers/Microsoft.Resources/deployments/0e2d10aecc9dee2ed3db4bdb9b7ef1c04b366c5a329e9fc125a96e2435247e25/operationStatuses/08584266545338751491\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5930368-svc/providers/Microsoft.Resources/deployments/cluster-oxhidoqvk2hqy\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5930368-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5930368-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5930368-svc/providers/Microsoft.ContainerService/managedClusters/prow-j5930368-svc/agentPools/u64d8ds61\\\",\\r\\n                    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n                    \\\"details\\\": [\\r\\n                      {\\r\\n                        \\\"code\\\": \\\"QuotaExceeded\\\",\\r\\n                        \\\"message\\\": \\\"Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5930368-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8ds61-24727517-vmss failed. Operation could not be completed as it results in exceeding approved StandardDdsv6Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: WestUS3, Current Limit: 350, Current Usage: 344, Additional Required: 8, (Minimum) New Limit Required: 352. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22WestUS3%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22StandardDdsv6Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22StandardDdsv6Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\\\",\\r\\n                        \\\"details\\\": [\\r\\n                          {\\r\\n                            \\\"code\\\": \\\"OperationNotAllowed\\\",\\r\\n                            \\\"message\\\": \\\"Operation could not be completed as it results in exceeding approved StandardDdsv6Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: WestUS3, Current Limit: 350, Current Usage: 344, Additional Required: 8, (Minimum) New Limit Required: 352. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22WestUS3%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22StandardDdsv6Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22StandardDdsv6Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\\\"\\r\\n                          }\\r\\n                        ]\\r\\n                      }\\r\\n                    ]\\r\\n                  },\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5930368-svc/providers/Microsoft.ContainerService/managedClusters/prow-j5930368-svc/agentPools/u64d8ds62\\\",\\r\\n                    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n                    \\\"details\\\": [\\r\\n                      {\\r\\n                        \\\"code\\\": \\\"QuotaExceeded\\\",\\r\\n                        \\\"message\\\": \\\"Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5930368-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8ds62-48651519-vmss failed. Operation could not be completed as it results in exceeding approved StandardDdsv6Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: WestUS3, Current Limit: 350, Current Usage: 344, Additional Required: 8, (Minimum) New Limit Required: 352. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22WestUS3%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22StandardDdsv6Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22StandardDdsv6Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\\\",\\r\\n                        \\\"details\\\": [\\r\\n                          {\\r\\n                            \\\"code\\\": \\\"OperationNotAllowed\\\",\\r\\n                            \\\"message\\\": \\\"Operation could not be completed as it results in exceeding approved StandardDdsv6Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: WestUS3, Current Limit: 350, Current Usage: 344, Additional Required: 8, (Minimum) New Limit Required: 352. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22WestUS3%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22StandardDdsv6Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22StandardDdsv6Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\\\"\\r\\n                          }\\r\\n                        ]\\r\\n                      }\\r\\n                    ]\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-03-31T10:25:46.442Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-31T10:25:47.971Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-31T10:25:49.586Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=b9cb68ed03665f43cd7359a1a37db6e97726442e4f1c4d6116f6fab7c7405162
time=2026-03-31T10:40:22.734Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8256768-svc/providers/Microsoft.Resources/deployments/b9cb68ed03665f43cd7359a1a37db6e97726442e4f1c4d6116f6fab7c7405162/operationStatuses/08584266541368518434\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8256768-svc/providers/Microsoft.Resources/deployments/cluster-36zhw7t4wgyrw\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8256768-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8256768-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8256768-svc/providers/Microsoft.ContainerService/managedClusters/prow-j8256768-svc/agentPools/u64d8ds62\\\",\\r\\n                    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n                    \\\"details\\\": [\\r\\n                      {\\r\\n                        \\\"code\\\": \\\"QuotaExceeded\\\",\\r\\n                        \\\"message\\\": \\\"Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8256768-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8ds62-20084763-vmss failed. Operation could not be completed as it results in exceeding approved StandardDdsv6Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: WestUS3, Current Limit: 350, Current Usage: 344, Additional Required: 8, (Minimum) New Limit Required: 352. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22WestUS3%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22StandardDdsv6Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22StandardDdsv6Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\\\",\\r\\n                        \\\"details\\\": [\\r\\n                          {\\r\\n                            \\\"code\\\": \\\"OperationNotAllowed\\\",\\r\\n                            \\\"message\\\": \\\"Operation could not be completed as it results in exceeding approved StandardDdsv6Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: WestUS3, Current Limit: 350, Current Usage: 344, Additional Required: 8, (Minimum) New Limit Required: 352. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22WestUS3%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22StandardDdsv6Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22StandardDdsv6Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\\\"\\r\\n                          }\\r\\n                        ]\\r\\n                      }\\r\\n                    ]\\r\\n                  },\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8256768-svc/providers/Microsoft.ContainerService/managedClusters/prow-j8256768-svc/agentPools/u64d8ds61\\\",\\r\\n                    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n                    \\\"details\\\": [\\r\\n                      {\\r\\n                        \\\"code\\\": \\\"QuotaExceeded\\\",\\r\\n                        \\\"message\\\": \\\"Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8256768-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8ds61-36181724-vmss failed. Operation could not be completed as it results in exceeding approved StandardDdsv6Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: WestUS3, Current Limit: 350, Current Usage: 344, Additional Required: 8, (Minimum) New Limit Required: 352. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22WestUS3%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22StandardDdsv6Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22StandardDdsv6Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\\\",\\r\\n                        \\\"details\\\": [\\r\\n                          {\\r\\n                            \\\"code\\\": \\\"OperationNotAllowed\\\",\\r\\n                            \\\"message\\\": \\\"Operation could not be completed as it results in exceeding approved StandardDdsv6Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: WestUS3, Current Limit: 350, Current Usage: 344, Additional Required: 8, (Minimum) New Limit Required: 352. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22WestUS3%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22StandardDdsv6Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22StandardDdsv6Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\\\"\\r\\n                          }\\r\\n                        ]\\r\\n                      }\\r\\n                    ]\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-03-31T09:48:30.833Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-31T09:48:32.254Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-31T09:48:34.230Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=6a279a0961fa4de56873ef30419bc1a8e76cb111fa0e1d931976ca220b568aa5
time=2026-03-31T10:03:08.011Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j2284544-svc/providers/Microsoft.Resources/deployments/6a279a0961fa4de56873ef30419bc1a8e76cb111fa0e1d931976ca220b568aa5/operationStatuses/08584266563724128499\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2284544-svc/providers/Microsoft.Resources/deployments/cluster-3emv7dgipb3n4\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2284544-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2284544-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2284544-svc/providers/Microsoft.ContainerService/managedClusters/prow-j2284544-svc/agentPools/u64d8ds62\\\",\\r\\n                    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n                    \\\"details\\\": [\\r\\n                      {\\r\\n                        \\\"code\\\": \\\"QuotaExceeded\\\",\\r\\n                        \\\"message\\\": \\\"Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2284544-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8ds62-31665093-vmss failed. Operation could not be completed as it results in exceeding approved StandardDdsv6Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: WestUS3, Current Limit: 350, Current Usage: 344, Additional Required: 8, (Minimum) New Limit Required: 352. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22WestUS3%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22StandardDdsv6Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22StandardDdsv6Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\\\",\\r\\n                        \\\"details\\\": [\\r\\n                          {\\r\\n                            \\\"code\\\": \\\"OperationNotAllowed\\\",\\r\\n                            \\\"message\\\": \\\"Operation could not be completed as it results in exceeding approved StandardDdsv6Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: WestUS3, Current Limit: 350, Current Usage: 344, Additional Required: 8, (Minimum) New Limit Required: 352. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22WestUS3%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22StandardDdsv6Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22StandardDdsv6Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\\\"\\r\\n                          }\\r\\n                        ]\\r\\n                      }\\r\\n                    ]\\r\\n                  },\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2284544-svc/providers/Microsoft.ContainerService/managedClusters/prow-j2284544-svc/agentPools/u64d8ds61\\\",\\r\\n                    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n                    \\\"details\\\": [\\r\\n                      {\\r\\n                        \\\"code\\\": \\\"QuotaExceeded\\\",\\r\\n                        \\\"message\\\": \\\"Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2284544-svc-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-u64d8ds61-24371072-vmss failed. Operation could not be completed as it results in exceeding approved StandardDdsv6Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: WestUS3, Current Limit: 350, Current Usage: 344, Additional Required: 8, (Minimum) New Limit Required: 352. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22WestUS3%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22StandardDdsv6Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22StandardDdsv6Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\\\",\\r\\n                        \\\"details\\\": [\\r\\n                          {\\r\\n                            \\\"code\\\": \\\"OperationNotAllowed\\\",\\r\\n                            \\\"message\\\": \\\"Operation could not be completed as it results in exceeding approved StandardDdsv6Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: WestUS3, Current Limit: 350, Current Usage: 344, Additional Required: 8, (Minimum) New Limit Required: 352. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22WestUS3%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22StandardDdsv6Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22StandardDdsv6Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\\\"\\r\\n                          }\\r\\n                        ]\\r\\n                      }\\r\\n                    ]\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster	17

Affected runs (17)

Date (UTC)	Associated PR	Prow job
2026-03-31 10:12	PR #4702	2038922313855930368
2026-03-31 09:59	PR #4700	2038919046388256768
2026-03-31 09:43	PR #4702	2038915125842284544
2026-03-31 08:26	PR #4699	2038895766440775680
2026-03-31 07:57	PR #4671	2038888351062298624
2026-03-31 06:30	PR #4640	2038866390617690112
2026-03-31 00:37	PR #4457	2038777586141630464
2026-03-30 22:32	PR #4611	2038746232985227264
2026-03-30 21:36	PR #4693	2038732136319553536
2026-03-30 21:05	PR #4694	2038724210125180928
2026-03-30 19:49	PR #4610	2038705207952543744
2026-03-30 19:41	PR #4690	2038703268648980480
2026-03-30 18:54	PR #4693	2038691474815586304
2026-03-30 18:47	PR #4603	2038689609893810176
2026-03-30 18:33	PR #4610	2038686183726256128
2026-03-30 18:24	PR #4687	2038684224747212800
2026-03-30 17:16	PR #4692	2038667880081395712

Interrupted by User

full failure pattern:

Interrupted by User

Signal:

e2e

PROD

Full failure examples (3)

fail [:0]: Interrupted by User
fail [:0]: Interrupted by User

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:42]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:42]: Interrupted by User

Contributing tests (16)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled	6
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19	5
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21	5
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	4
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	4
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.19	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1

Affected runs (16)

Date (UTC)	Associated PR	Prow job
2026-04-02 16:37	PR #4723	2039744162546323456
2026-04-01 09:29	PR #4664	2039274042879381504
2026-04-01 09:29	PR #4670	2039273942891368448
2026-04-01 09:28	PR #4673	2039273635952201728
2026-04-01 09:27	PR #4693	2039273320041418752
2026-04-01 09:26	PR #4695	2039273285383884800
2026-04-01 09:26	PR #4696	2039273255478497280
2026-04-01 09:26	PR #4702	2039273164030087168
2026-04-01 09:26	PR #4707	2039273076775981056
2026-04-01 09:25	PR #4708	2039273039144685568
2026-04-01 09:25	PR #4709	2039273036514856960
2026-04-01 09:25	PR #4710	2039273001324646400
2026-04-01 09:25	PR #4712	2039272919170813952
2026-04-01 09:25	PR #4714	2039272851831263232
2026-03-30 23:43	PR #4695	2038764111487045632
2026-03-30 17:50	PR #4688	2038675791503429632

failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...

full failure pattern:

failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000e941e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-gpk4bh" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0007fb840>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-gpk4bh\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0007fb820>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-gpk4bh\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0007fb800>{
                        msg: "failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-gpk4bh\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xbee3c60>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000e941e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-gpk4bh" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0007fb840>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-gpk4bh\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0007fb820>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-gpk4bh\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0007fb800>{
                        msg: "failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-gpk4bh\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xbee3c60>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001588228>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-rvx4rm" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000d870e0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-rvx4rm\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000d870c0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-rvx4rm\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000d870a0>{
                        msg: "failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-rvx4rm\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xbee3c60>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001588228>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-rvx4rm" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000d870e0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-rvx4rm\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000d870c0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-rvx4rm\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000d870a0>{
                        msg: "failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-rvx4rm\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xbee3c60>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000b92558>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-lrrz7b" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0015d9140>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-lrrz7b\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0015d9120>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-lrrz7b\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0015d9100>{
                        msg: "failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-lrrz7b\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xbee3c60>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000b92558>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-lrrz7b" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0015d9140>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-lrrz7b\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0015d9120>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-lrrz7b\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0015d9100>{
                        msg: "failed waiting for hcpCluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-lrrz7b\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xbee3c60>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred

Contributing tests (23)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	4
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	4
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.19	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to log into a cluster via a breakglass session	1

Affected runs (14)

Date (UTC)	Associated PR	Prow job
2026-04-03 06:49	PR #4720	2039958427727302656
2026-04-02 10:00	PR #4652	2039644149858177024
2026-04-02 05:15	PR #4730	2039572528229257216
2026-04-01 10:09	PR #4661	2039284079249068032
2026-04-01 09:29	PR #4670	2039273942891368448
2026-04-01 09:28	PR #4673	2039273635952201728
2026-04-01 09:27	PR #4693	2039273320041418752
2026-04-01 09:26	PR #4696	2039273255478497280
2026-04-01 09:26	PR #4702	2039273164030087168
2026-04-01 09:25	PR #4708	2039273039144685568
2026-04-01 09:25	PR #4647	2039273018135416832
2026-04-01 09:25	PR #4710	2039273001324646400
2026-04-01 09:25	PR #4712	2039272919170813952
2026-04-01 09:25	PR #4714	2039272851831263232

ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid

full failure pattern:

ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid

Signal:

provision

none

Full failure examples (3)

time=2026-04-02T19:48:00.107Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-04-02T19:48:02.240Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-04-02T19:48:03.698Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=0c4911c1685a20609549f75db7625cacf2320f36305028898d025569bae2fa4c
time=2026-04-02T19:49:34.390Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j2699776/providers/Microsoft.Resources/deployments/0c4911c1685a20609549f75db7625cacf2320f36305028898d025569bae2fa4c/operationStatuses/08584264476026849074\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2699776/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2699776/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j2699776\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report c42af2c3-7d12-4171-9bd0-571348450d45:4/2/2026 7:49:23 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-04-02T09:42:00.307Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-04-02T09:42:02.470Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-04-02T09:42:03.710Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=63e389fe98fd449bd7e9b47c13ebfac931288071ccae0047c27b89b43672b53b
time=2026-04-02T09:45:04.543Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j2951808/providers/Microsoft.Resources/deployments/63e389fe98fd449bd7e9b47c13ebfac931288071ccae0047c27b89b43672b53b/operationStatuses/08584264839624194560\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2951808/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2951808/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j2951808\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 899bf80f-8b91-429f-98ec-271a3fd806ee:4/2/2026 9:43:24 AM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-03-31T21:49:47.164Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-31T21:49:49.145Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-31T21:49:50.818Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=5666de01ff990fe7d2ae6567560287fa6382c5ae27f33e147785947a49ead9c2
time=2026-03-31T21:51:21.528Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j2563072/providers/Microsoft.Resources/deployments/5666de01ff990fe7d2ae6567560287fa6382c5ae27f33e147785947a49ead9c2/operationStatuses/08584266130954832843\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2563072/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2563072/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j2563072\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report f0268d91-3185-4228-8846-92fac12c20b5:3/31/2026 9:51:01 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra	7

Affected runs (7)

Date (UTC)	Associated PR	Prow job
2026-04-02 19:21	PR #4393	2039785313722699776
2026-04-02 09:21	PR #4705	2039634347752951808
2026-03-31 21:35	PR #4626	2039094380882563072
2026-03-31 12:33	PR #4630	2038957769591099392
2026-03-31 07:16	PR #4697	2038878094260965376
2026-03-30 15:02	PR #4257	2038632945115205632
2026-03-30 10:03	PR #4674	2038557755538673664

resource not ready, name: maestro-db, kind: PersistentVolumeClaim, status: InProgress resource not ready, name: maestro, kind: Deployment, status: InProgress resource not ready, n...

full failure pattern:

resource not ready, name: maestro-db, kind: PersistentVolumeClaim, status: InProgress resource not ready, name: maestro, kind: Deployment, status: InProgress resource not ready, name: maestro-db, kind: Deployment, status: InProgress context deadline exceeded

Signal:

provision

none

Full failure examples (3)

time=2026-03-31T21:13:25.245Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-31T21:13:25.866Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy values="map[azure:map[clientId:87ac7b71-7b47-4dbb-82e5-fe4f9689c7dc cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] backend:map[namespace:aro-hcp serviceAccount:backend] broker:map[host:arohcp-prow-maestro-j1862144.westus3-1.ts.eventgrid.azure.net port:8883] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:maestro-server-j1862144] database:map[azureDb:false containerizedDbImage:docker.io/library/postgres:14.2 containerizedDbPvcCapacity:512Mi debug:false host:maestro-db maxOpenConnections:50 name:maestro password:TheBlurstOfTimes port:5432 user:maestro-server] deployment:map[replicas:3 resources:map[limits:map[memory:unlimited] requests:map[cpu:200m memory:512Mi]] zoneCount:4] image:map[digest:sha256:195db1fb826c90b6f3a7621b2d11ed79d632e4e93a5dc475fb4e81d2c2e44cae registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] istio:map[restrictIngress:true] maestro:map[glog_v:2 grpcBindPort:8090 healthCheckBindPort:8083 httpBindPort:8000 httpReadTimeout:60s httpWriteTimeout:60s labelMetricsInclusionDuration:168h serviceAccount:maestro] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro:pull workloadIdentityClientId:9a580b67-a656-4b07-a2b3-837e48454759 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-31T21:13:25.866Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T21:13:26.134Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy namespace=maestro
time=2026-03-31T21:13:26.478Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy dryRun=false
time=2026-03-31T21:13:26.478Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T21:13:26.675Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T21:18:33.900Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy err="resource not ready, name: maestro-db, kind: PersistentVolumeClaim, status: InProgress\nresource not ready, name: maestro, kind: Deployment, status: InProgress\nresource not ready, name: maestro-db, kind: Deployment, status: InProgress\ncontext deadline exceeded"
time=2026-03-31T21:18:33.900Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T21:18:33.900Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T21:18:38.206Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy release=maestro-server namespace=maestro status=failed description="Release \"maestro-server\" failed: resource not ready, name: maestro-db, kind: PersistentVolumeClaim, status: InProgress\nresource not ready, name: maestro, kind: Deployment, status: InProgress\nresource not ready, name: maestro-db, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[azure:map[clientId:87ac7b71-7b47-4dbb-82e5-fe4f9689c7dc cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] backend:map[namespace:aro-hcp serviceAccount:backend] broker:map[host:arohcp-prow-maestro-j1862144.westus3-1.ts.eventgrid.azure.net port:8883] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:maestro-server-j1862144] database:map[azureDb:false containerizedDbImage:docker.io/library/postgres:14.2 containerizedDbPvcCapacity:512Mi debug:false host:maestro-db maxOpenConnections:50 name:maestro password:TheBlurstOfTimes port:5432 user:maestro-server] deployment:map[replicas:3 resources:map[limits:map[memory:unlimited] requests:map[cpu:200m memory:512Mi]] zoneCount:4] image:map[digest:sha256:195db1fb826c90b6f3a7621b2d11ed79d632e4e93a5dc475fb4e81d2c2e44cae registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] istio:map[restrictIngress:true] maestro:map[glog_v:2 grpcBindPort:8090 healthCheckBindPort:8083 httpBindPort:8000 httpReadTimeout:60s httpWriteTimeout:60s labelMetricsInclusionDuration:168h serviceAccount:maestro] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro:pull workloadIdentityClientId:9a580b67-a656-4b07-a2b3-837e48454759 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-31T21:18:38.208Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5yUQY/aPBCG7/kVo5xAIqsFpNUnPu2BbnuqWiG16mGrHhz7bWJw7Gg8AVHx46tAt1AaINtbMn6e0czYmsRBiBFDwxqRHskoUaJyh0HIl9Dy3nozi8LWFyM6hD6qCr9DWMPvI7FW+iU8/JoQEaVvUbuwreAlHVFaKUTh8MfnFS4z+U20DlEKRsxyhloVTsXY4SwAnjdSwovVSmzwLWTwXTVOOvhP0AxZcFhbA376K2uHwGur8SF4K4FPe4jgNfiyMtc6NLfGc6jnlUz3+BbgaKPAy5fgmgpPTtnqpvWr2FOu4Fr3I0soJ6UuoVe9hH5ZKwhb3XXbHXBnU3PNi8a5N9Yb64v9a2qcy/Lj/7nQSBnY/ti/oEVwVm9bSjkXNlmu9ArevFbTrokC3r+T8zH00F8alEs9XpUvjvCq5oOU/zAgRmGj8P4sW4a8rbnjxpNv/yerJgd7COK7drvEZEebEgwSWyGKqmrKIRvA08AoQRsepJP7yUN2P82m48+T8Ww8nU0entMh3d3RRea/2XT6nA6HyY6WwXpaWW8erffgk50YPB1X4ekOPF9+yY5qDu3psdAR1cEc4H5JRuRCMaIKMaqiTRnYgCnfnjRvEPXPAAAA//8MqjdpuQUAAA=="
time=2026-03-31T21:18:38.210Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T21:18:38.211Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: maestro-db, kind: PersistentVolumeClaim, status: InProgress\nresource not ready, name: maestro, kind: Deployment, status: InProgress\nresource not ready, name: maestro-db, kind: Deployment, status: InProgress\ncontext deadline exceeded"

time=2026-03-31T19:11:06.805Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-31T19:11:07.594Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy values="map[azure:map[clientId:dea65e2b-e96b-4a1f-8167-96dc478860d3 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] backend:map[namespace:aro-hcp serviceAccount:backend] broker:map[host:arohcp-prow-maestro-j8009600.westus3-1.ts.eventgrid.azure.net port:8883] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:maestro-server-j8009600] database:map[azureDb:false containerizedDbImage:docker.io/library/postgres:14.2 containerizedDbPvcCapacity:512Mi debug:false host:maestro-db maxOpenConnections:50 name:maestro password:TheBlurstOfTimes port:5432 user:maestro-server] deployment:map[replicas:3 resources:map[limits:map[memory:unlimited] requests:map[cpu:200m memory:512Mi]] zoneCount:4] image:map[digest:sha256:195db1fb826c90b6f3a7621b2d11ed79d632e4e93a5dc475fb4e81d2c2e44cae registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] istio:map[restrictIngress:true] maestro:map[glog_v:2 grpcBindPort:8090 healthCheckBindPort:8083 httpBindPort:8000 httpReadTimeout:60s httpWriteTimeout:60s labelMetricsInclusionDuration:168h serviceAccount:maestro] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro:pull workloadIdentityClientId:eb9ce891-c224-4a01-a584-07addfe85d82 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-31T19:11:07.594Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T19:11:07.915Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy namespace=maestro
time=2026-03-31T19:11:08.326Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy dryRun=false
time=2026-03-31T19:11:08.326Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T19:11:08.467Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T19:16:15.145Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy err="resource not ready, name: maestro-db, kind: PersistentVolumeClaim, status: InProgress\nresource not ready, name: maestro, kind: Deployment, status: InProgress\nresource not ready, name: maestro-db, kind: Deployment, status: InProgress\ncontext deadline exceeded"
time=2026-03-31T19:16:15.145Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T19:16:15.145Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T19:16:19.614Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy release=maestro-server namespace=maestro status=failed description="Release \"maestro-server\" failed: resource not ready, name: maestro-db, kind: PersistentVolumeClaim, status: InProgress\nresource not ready, name: maestro, kind: Deployment, status: InProgress\nresource not ready, name: maestro-db, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[azure:map[clientId:dea65e2b-e96b-4a1f-8167-96dc478860d3 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] backend:map[namespace:aro-hcp serviceAccount:backend] broker:map[host:arohcp-prow-maestro-j8009600.westus3-1.ts.eventgrid.azure.net port:8883] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:maestro-server-j8009600] database:map[azureDb:false containerizedDbImage:docker.io/library/postgres:14.2 containerizedDbPvcCapacity:512Mi debug:false host:maestro-db maxOpenConnections:50 name:maestro password:TheBlurstOfTimes port:5432 user:maestro-server] deployment:map[replicas:3 resources:map[limits:map[memory:unlimited] requests:map[cpu:200m memory:512Mi]] zoneCount:4] image:map[digest:sha256:195db1fb826c90b6f3a7621b2d11ed79d632e4e93a5dc475fb4e81d2c2e44cae registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] istio:map[restrictIngress:true] maestro:map[glog_v:2 grpcBindPort:8090 healthCheckBindPort:8083 httpBindPort:8000 httpReadTimeout:60s httpWriteTimeout:60s labelMetricsInclusionDuration:168h serviceAccount:maestro] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro:pull workloadIdentityClientId:eb9ce891-c224-4a01-a584-07addfe85d82 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-31T19:16:19.616Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5yUwW7TQBCG736KkU+JFFdJKyoI6qEUTggUqYhDEYf17o+9yXrXmh0nCsrDI6cqcUOcmt42s983mp2JJ3EQYsTQsEakGzJKlKjcYRTyJbR8tt7Mo7D1xYQeQ19Vhb8hrOH3kVgr/RQe/0iIiNJ7aIakE0orhSgcnh17mMzkJ7AFONoo8PI9uKbCnVO2etG6B6+tRpcruNbDyBLKSalL6NUgYVjWCsJWx2HwuUd9Cd5K4C4ewWtwv3KrdWj8CwP5iNqFbYVXcacrfo7WIUrBiFnOUKvCqXiqG7eaF41zH6w31hd7r3Euyw+/j4VGysD2txIb/CI4q7ctpZwLmyxXegVv/lfTrokC3jf2eNgD9KemSE9fzsu9f5Szmg9SvqJBjMJG4f1dtgx5W3P/7BcAt+ngxeq900IGv1TjpPcjX3BYWwO++2fkh2Py832yanKwhyB+ardLTHa0KcEgsRWiqKqmHLIBPI2MErThUXo5vbzOplfZ1ezb7N18NptP3z6kY7q4oF7mej5785COx8mOlsF6Wllvbqz34M5ODJ4Oq7C7A4+XX7KjmkN7eyh0QnUwj/CwJBNyoZhQhRhV0aYMbMCUbzuPN4j6TwAAAP//CEKyyLkFAAA="
time=2026-03-31T19:16:19.618Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T19:16:19.618Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: maestro-db, kind: PersistentVolumeClaim, status: InProgress\nresource not ready, name: maestro, kind: Deployment, status: InProgress\nresource not ready, name: maestro-db, kind: Deployment, status: InProgress\ncontext deadline exceeded"

time=2026-03-31T18:15:49.871Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-31T18:15:50.601Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy values="map[azure:map[clientId:f6ba2ba4-0ff8-4583-ac1c-bb4e8dde3733 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] backend:map[namespace:aro-hcp serviceAccount:backend] broker:map[host:arohcp-prow-maestro-j3484800.westus3-1.ts.eventgrid.azure.net port:8883] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:maestro-server-j3484800] database:map[azureDb:false containerizedDbImage:docker.io/library/postgres:14.2 containerizedDbPvcCapacity:512Mi debug:false host:maestro-db maxOpenConnections:50 name:maestro password:TheBlurstOfTimes port:5432 user:maestro-server] deployment:map[replicas:3 resources:map[limits:map[memory:unlimited] requests:map[cpu:200m memory:512Mi]] zoneCount:4] image:map[digest:sha256:195db1fb826c90b6f3a7621b2d11ed79d632e4e93a5dc475fb4e81d2c2e44cae registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] istio:map[restrictIngress:true] maestro:map[glog_v:2 grpcBindPort:8090 healthCheckBindPort:8083 httpBindPort:8000 httpReadTimeout:60s httpWriteTimeout:60s labelMetricsInclusionDuration:168h serviceAccount:maestro] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro:pull workloadIdentityClientId:66c55af7-9abb-42e4-8590-e24201076010 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-31T18:15:50.601Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T18:15:50.918Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy namespace=maestro
time=2026-03-31T18:15:51.359Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy dryRun=false
time=2026-03-31T18:15:51.359Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T18:15:51.585Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T18:20:59.038Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy err="resource not ready, name: maestro-db, kind: PersistentVolumeClaim, status: InProgress\nresource not ready, name: maestro, kind: Deployment, status: InProgress\nresource not ready, name: maestro-db, kind: Deployment, status: InProgress\ncontext deadline exceeded"
time=2026-03-31T18:20:59.038Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T18:20:59.038Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T18:21:03.502Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy release=maestro-server namespace=maestro status=failed description="Release \"maestro-server\" failed: resource not ready, name: maestro-db, kind: PersistentVolumeClaim, status: InProgress\nresource not ready, name: maestro, kind: Deployment, status: InProgress\nresource not ready, name: maestro-db, kind: Deployment, status: InProgress\ncontext deadline exceeded" values="map[azure:map[clientId:f6ba2ba4-0ff8-4583-ac1c-bb4e8dde3733 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] backend:map[namespace:aro-hcp serviceAccount:backend] broker:map[host:arohcp-prow-maestro-j3484800.westus3-1.ts.eventgrid.azure.net port:8883] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:maestro-server-j3484800] database:map[azureDb:false containerizedDbImage:docker.io/library/postgres:14.2 containerizedDbPvcCapacity:512Mi debug:false host:maestro-db maxOpenConnections:50 name:maestro password:TheBlurstOfTimes port:5432 user:maestro-server] deployment:map[replicas:3 resources:map[limits:map[memory:unlimited] requests:map[cpu:200m memory:512Mi]] zoneCount:4] image:map[digest:sha256:195db1fb826c90b6f3a7621b2d11ed79d632e4e93a5dc475fb4e81d2c2e44cae registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] istio:map[restrictIngress:true] maestro:map[glog_v:2 grpcBindPort:8090 healthCheckBindPort:8083 httpBindPort:8000 httpReadTimeout:60s httpWriteTimeout:60s labelMetricsInclusionDuration:168h serviceAccount:maestro] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro:pull workloadIdentityClientId:66c55af7-9abb-42e4-8590-e24201076010 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-03-31T18:21:03.505Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5yUQY/TMBCF7/kVo5y2UrNquyqCoj0sCycEqgTisIiDYz8St44djSetivrjUboqDVXTDdzayfeePWP7JQ5CjBga1oh0T0aJEpU73IR8BS0frTeLKGx9Mabn0mdV4U8JG/hDJdZKH8uj7wkRUfoFmiFLDhtrwI9OxZiOKa0UonD46+dRwBur8Sl4K4E7QBbBG3C/5EHr0Hi5br8ERxsFXr4F11R4dMpW3VVM3r9Clyu41sPIEspJqUvo9SDBMNcKwlbHYfDFph40Lxvn3llvrC9aoG6cy/LT/3NBI2Vg+0uJDX4ZnNW7llLOhW2WK72GN/8q066JAj4c7vkYBsiPDUpfj1fFvSO8KvNByv8YEKOwUfjwLVuFvN3ztXsKbu3gxeqDpoUMfqrGycVDb9/ZS7fojLk8s/eoXdhVeOkpXeSGWNYhSsGIWc5Q6+I8E5Ifb5N1k4M9BPFDmy4x2dO2BIPEVoiiqppyyBbwdGOUoC3fpLPJ7FU2ucvupl+nrxfT+WI+fUpHdHtLfcxsspi/eUpHo2RPq2A9ra0399Z7cCcTg6dTFHYz8Dz8kj3VHNqvp42OqQ7mGR5mMiYXijFViFEVrWVgA6Z812neIOrfAQAA///EFa4XuQUAAA=="
time=2026-03-31T18:21:03.508Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy
time=2026-03-31T18:21:03.508Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: maestro-db, kind: PersistentVolumeClaim, status: InProgress\nresource not ready, name: maestro, kind: Deployment, status: InProgress\nresource not ready, name: maestro-db, kind: Deployment, status: InProgress\ncontext deadline exceeded"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Maestro.Server/service/deploy	5

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-03-31 20:32	PR #4690	2039078410201862144
2026-03-31 18:46	PR #4690	2039051671258009600
2026-03-31 17:36	PR #4690	2039033987233484800
2026-03-31 13:47	PR #4690	2038976452795305984
2026-03-30 17:51	PR #4690	2038677165528059904

Deserializaion Error: no output from command

full failure pattern:

Deserializaion Error: no output from command

Signal:

e2e

PROD

Full failure examples (3)

HCP/test/util/framework/hcp_helper.go:201 +0x218 fp=0xc0000dbef0 sp=0xc0000dbde8 pc=0x525c458
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:349 +0x94 fp=0xc0000dbf78 sp=0xc0000dbef0 pc=0x525ea54
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc0000dbfe0 sp=0xc0000dbf78 pc=0x3af0ab0
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000dbfe8 sp=0xc0000dbfe0 pc=0x2dbecc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 850
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

goroutine 861 gp=0xc000168000 m=nil [IO wait]:
runtime.gopark(0xc0009da6c0?, 0x3100000000?, 0x50?, 0x7?, 0xb?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000c70720 sp=0xc000c70700 pc=0x2db650e
runtime.netpollblock(0x2e20d18?, 0x2d42066?, 0x0?)
	/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc000c70758 sp=0xc000c70720 pc=0x2d75fb7
internal/poll.runtime_pollWait(0x7f75e94bf600, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc000c70778 sp=0xc000c70758 pc=0x2db4da5
internal/poll.(*pollDesc).wait(0xc00050e500?, 0xc00147b000?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000c707a0 sp=0xc000c70778 pc=0x2e43a47
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00050e500, {0xc00147b000, 0x3000, 0x3000})
	/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc000c70838 sp=0xc000c707a0 pc=0x2e44e79
net.(*netFD).Read(0xc00050e500, {0xc00147b000?, 0xf?, 0xc000cd8750?})
	/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc000c70880 sp=0xc000c70838 pc=0x300a4e5
net.(*conn).Read(0xc00019cb68, {0xc00147b000?, 0x7f75e8391040?, 0x7f76318eaf30?})
	/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc000c708c8 sp=0xc000c70880 pc=0x301c585
crypto/tls.(*atLeastReader).Read(0xc0010805a0, {0xc00147b000?, 0xc001440b01?, 0xf?})
	/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc000c70910 sp=0xc000c708c8 pc=0x308801b
bytes.(*Buffer).ReadFrom(0xc00075c9a8, {0x8c14100, 0xc0010805a0})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc000c70968 sp=0xc000c70910 pc=0x2e81c78
crypto/tls.(*Conn).readFromUntil(0xc00075c708, {0x8c14ee0, 0xc00019cb68}, 0xc000c709d0?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc000c709a0 sp=0xc000c70968 pc=0x30881fe
crypto/tls.(*Conn).readRecordOrCCS(0xc00075c708, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc000c70c18 sp=0xc000c709a0 pc=0x308527b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc00075c708, {0xc000e62000, 0x1000, 0x34ca980?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc000c70c80 sp=0xc000c70c18 pc=0x308bb65
bufio.(*Reader).Read(0xc000fe4600, {0xc000948584, 0x9, 0x34d8a2e?})
	/usr/lib/golang/src/bufio/bufio.go:245 +0x197 fp=0xc000c70cb8 sp=0xc000c70c80 pc=0x2f1e1d7
io.ReadAtLeast({0x8c13700, 0xc000fe4600}, {0xc000948584, 0x9, 0x9}, 0x9)
	/usr/lib/golang/src/io/io.go:335 +0x8e fp=0xc000c70d00 sp=0xc000c70cb8 pc=0x2e10e8e
io.ReadFull(...)
	/usr/lib/golang/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc000948584, 0x9, 0xc00000007b?}, {0x8c13700?, 0xc000fe4600?})
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:250 +0x65 fp=0xc000c70d50 sp=0xc000c70d00 pc=0x34c1745
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc000948540)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:513 +0x6b fp=0xc000c70dd8 sp=0xc000c70d50 pc=0x34c230b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000948540)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:572 +0x18 fp=0xc000c70e18 sp=0xc000c70dd8 pc=0x34c2a78
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000c70fa8)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2254 +0xca fp=0xc000c70f60 sp=0xc000c70e18 pc=0x34d596a
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00097ee00)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2123 +0x52 fp=0xc000c70fc8 sp=0xc000c70f60 pc=0x34d4cf2
golang.org/x/net/http2.(*Transport).newClientConn.gowrap1()
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0x25 fp=0xc000c70fe0 sp=0xc000c70fc8 pc=0x34ce485
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000c70fe8 sp=0xc000c70fe0 pc=0x2dbecc1
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 860
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0xda5

rax    0xca
rbx    0x0
rcx    0x2dc0ae3
rdx    0x0
rdi    0xc1bd5b0
rsi    0x80
rbp    0x7ffdf970e738
rsp    0x7ffdf970e6f0
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x7ffdf970e718
r13    0xc0000cf490
r14    0xc1bb4c0
r15    0xffffffffffffffff
rip    0x2dc0ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
HCP/test/util/framework/hcp_helper.go:201 +0x218 fp=0xc0000dbef0 sp=0xc0000dbde8 pc=0x525c458
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:349 +0x94 fp=0xc0000dbf78 sp=0xc0000dbef0 pc=0x525ea54
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc0000dbfe0 sp=0xc0000dbf78 pc=0x3af0ab0
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000dbfe8 sp=0xc0000dbfe0 pc=0x2dbecc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 850
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

goroutine 861 gp=0xc000168000 m=nil [IO wait]:
runtime.gopark(0xc0009da6c0?, 0x3100000000?, 0x50?, 0x7?, 0xb?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000c70720 sp=0xc000c70700 pc=0x2db650e
runtime.netpollblock(0x2e20d18?, 0x2d42066?, 0x0?)
	/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc000c70758 sp=0xc000c70720 pc=0x2d75fb7
internal/poll.runtime_pollWait(0x7f75e94bf600, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc000c70778 sp=0xc000c70758 pc=0x2db4da5
internal/poll.(*pollDesc).wait(0xc00050e500?, 0xc00147b000?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000c707a0 sp=0xc000c70778 pc=0x2e43a47
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00050e500, {0xc00147b000, 0x3000, 0x3000})
	/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc000c70838 sp=0xc000c707a0 pc=0x2e44e79
net.(*netFD).Read(0xc00050e500, {0xc00147b000?, 0xf?, 0xc000cd8750?})
	/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc000c70880 sp=0xc000c70838 pc=0x300a4e5
net.(*conn).Read(0xc00019cb68, {0xc00147b000?, 0x7f75e8391040?, 0x7f76318eaf30?})
	/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc000c708c8 sp=0xc000c70880 pc=0x301c585
crypto/tls.(*atLeastReader).Read(0xc0010805a0, {0xc00147b000?, 0xc001440b01?, 0xf?})
	/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc000c70910 sp=0xc000c708c8 pc=0x308801b
bytes.(*Buffer).ReadFrom(0xc00075c9a8, {0x8c14100, 0xc0010805a0})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc000c70968 sp=0xc000c70910 pc=0x2e81c78
crypto/tls.(*Conn).readFromUntil(0xc00075c708, {0x8c14ee0, 0xc00019cb68}, 0xc000c709d0?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc000c709a0 sp=0xc000c70968 pc=0x30881fe
crypto/tls.(*Conn).readRecordOrCCS(0xc00075c708, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc000c70c18 sp=0xc000c709a0 pc=0x308527b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc00075c708, {0xc000e62000, 0x1000, 0x34ca980?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc000c70c80 sp=0xc000c70c18 pc=0x308bb65
bufio.(*Reader).Read(0xc000fe4600, {0xc000948584, 0x9, 0x34d8a2e?})
	/usr/lib/golang/src/bufio/bufio.go:245 +0x197 fp=0xc000c70cb8 sp=0xc000c70c80 pc=0x2f1e1d7
io.ReadAtLeast({0x8c13700, 0xc000fe4600}, {0xc000948584, 0x9, 0x9}, 0x9)
	/usr/lib/golang/src/io/io.go:335 +0x8e fp=0xc000c70d00 sp=0xc000c70cb8 pc=0x2e10e8e
io.ReadFull(...)
	/usr/lib/golang/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc000948584, 0x9, 0xc00000007b?}, {0x8c13700?, 0xc000fe4600?})
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:250 +0x65 fp=0xc000c70d50 sp=0xc000c70d00 pc=0x34c1745
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc000948540)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:513 +0x6b fp=0xc000c70dd8 sp=0xc000c70d50 pc=0x34c230b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000948540)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:572 +0x18 fp=0xc000c70e18 sp=0xc000c70dd8 pc=0x34c2a78
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000c70fa8)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2254 +0xca fp=0xc000c70f60 sp=0xc000c70e18 pc=0x34d596a
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00097ee00)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2123 +0x52 fp=0xc000c70fc8 sp=0xc000c70f60 pc=0x34d4cf2
golang.org/x/net/http2.(*Transport).newClientConn.gowrap1()
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0x25 fp=0xc000c70fe0 sp=0xc000c70fc8 pc=0x34ce485
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000c70fe8 sp=0xc000c70fe0 pc=0x2dbecc1
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 860
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0xda5

rax    0xca
rbx    0x0
rcx    0x2dc0ae3
rdx    0x0
rdi    0xc1bd5b0
rsi    0x80
rbp    0x7ffdf970e738
rsp    0x7ffdf970e6f0
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x7ffdf970e718
r13    0xc0000cf490
r14    0xc1bb4c0
r15    0xffffffffffffffff
rip    0x2dc0ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command

ntext).CleanupResourceGroups(0xc000b203c0, {0x7fc2e6f48e08, 0xc001061f50}, {{0xc00096cf70, 0x1, 0x1}, 0x34630b8a000, {0x81bd64c, 0x8}})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:308 +0x273 fp=0xc00181f768 sp=0xc00181f658 pc=0x52794d3
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources(0xc000b203c0, {0x7fc2e6f48e08, 0xc001061f50})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:227 +0x225 fp=0xc00181f888 sp=0xc00181f768 pc=0x5278d65
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources-fm({0x7fc2e6f48e08?, 0xc001061f50?})
	<autogenerated>:1 +0x33 fp=0xc00181f8b0 sp=0xc00181f888 pc=0x52879f3
runtime.call16(0xc002041440, 0xc000a808c0, 0x0, 0x0, 0x0, 0x10, 0xc00181fb08)
	/usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc00181f8d0 sp=0xc00181f8b0 pc=0x2dbc223
runtime.reflectcall(0x7bba7a0?, 0xc001061f50?, 0x2?, 0x81f159f?, 0x0?, 0x12?, 0x77ba800?)
	<autogenerated>:1 +0x36 fp=0xc00181f910 sp=0xc00181f8d0 pc=0x2dc4ab6
reflect.Value.call({0x70c8de0?, 0xc000a808c0?, 0xc00151f4e0?}, {0x81b1730, 0x4}, {0xc001817f60, 0x1, 0x9?})
	/usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc00181fe70 sp=0xc00181f910 pc=0x2df7a86
reflect.Value.Call({0x70c8de0?, 0xc000a808c0?, 0xc000a5f618?}, {0xc001817f60?, 0x53b091d?, 0x2db550e?})
	/usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc00181fee8 sp=0xc00181fe70 pc=0x2df6b79
github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x8cbb3a0?, 0xc001061f50?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc00181ff88 sp=0xc00181fee8 pc=0x3219ff0
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc00181ffe0 sp=0xc00181ff88 pc=0x322cabb
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00181ffe8 sp=0xc00181ffe0 pc=0x2dbdcc1
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 91
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

goroutine 582 gp=0xc001156fc0 m=nil [sync.WaitGroup.Wait, 39 minutes]:
runtime.gopark(0xc00152faf0?, 0x2d4c874?, 0xe0?, 0x70?, 0x7fc2e6f99840?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000e37a90 sp=0xc000e37a70 pc=0x2db550e
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:466
runtime.semacquire1(0xc000ba0110, 0x0, 0x1, 0x0, 0x19)
	/usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc000e37af8 sp=0xc000e37a90 pc=0x2d9159d
sync.runtime_SemacquireWaitGroup(0xc001e214d0?, 0xc0?)
	/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc000e37b30 sp=0xc000e37af8 pc=0x2db6c2e
sync.(*WaitGroup).Wait(0xc000ba0108)
	/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc000e37b58 sp=0xc000e37b30 pc=0x2dda1a5
golang.org/x/sync/errgroup.(*Group).Wait(0xc000ba0100)
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:56 +0x1e fp=0xc000e37b78 sp=0xc000e37b58 pc=0x3aef95e
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters({0x7fc2e6f48e08, 0xc001061f50}, 0xc00116a780, {0xc000a94e88, 0x11}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:352 +0x8ef fp=0xc000e37e20 sp=0xc000e37b78 pc=0x525d7af
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).cleanupResourceGroup(0xc000b203c0, {0x7fc2e6f48e08, 0xc001061f50}, {0xc000a94e88, 0x11}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:436 +0x294 fp=0xc000e37f28 sp=0xc000e37e20 pc=0x527aed4
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7fc2e6f48e08, 0xc001061f50})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:297 +0x14d fp=0xc000e37fc0 sp=0xc000e37f28 pc=0x5279a0d
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:306 +0x28 fp=0xc000e37fe0 sp=0xc000e37fc0 pc=0x5279888
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000e37fe8 sp=0xc000e37fe0 pc=0x2dbdcc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 581
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:290 +0xc5

rax    0xca
rbx    0x0
rcx    0x2dbfae3
rdx    0x0
rdi    0xc1aa530
rsi    0x80
rbp    0x7ffdfbbb2ca0
rsp    0x7ffdfbbb2c58
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x581c
r13    0x1
r14    0xc1a8440
r15    0x1
rip    0x2dbfae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
ntext).CleanupResourceGroups(0xc000b203c0, {0x7fc2e6f48e08, 0xc001061f50}, {{0xc00096cf70, 0x1, 0x1}, 0x34630b8a000, {0x81bd64c, 0x8}})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:308 +0x273 fp=0xc00181f768 sp=0xc00181f658 pc=0x52794d3
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources(0xc000b203c0, {0x7fc2e6f48e08, 0xc001061f50})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:227 +0x225 fp=0xc00181f888 sp=0xc00181f768 pc=0x5278d65
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources-fm({0x7fc2e6f48e08?, 0xc001061f50?})
	<autogenerated>:1 +0x33 fp=0xc00181f8b0 sp=0xc00181f888 pc=0x52879f3
runtime.call16(0xc002041440, 0xc000a808c0, 0x0, 0x0, 0x0, 0x10, 0xc00181fb08)
	/usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc00181f8d0 sp=0xc00181f8b0 pc=0x2dbc223
runtime.reflectcall(0x7bba7a0?, 0xc001061f50?, 0x2?, 0x81f159f?, 0x0?, 0x12?, 0x77ba800?)
	<autogenerated>:1 +0x36 fp=0xc00181f910 sp=0xc00181f8d0 pc=0x2dc4ab6
reflect.Value.call({0x70c8de0?, 0xc000a808c0?, 0xc00151f4e0?}, {0x81b1730, 0x4}, {0xc001817f60, 0x1, 0x9?})
	/usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc00181fe70 sp=0xc00181f910 pc=0x2df7a86
reflect.Value.Call({0x70c8de0?, 0xc000a808c0?, 0xc000a5f618?}, {0xc001817f60?, 0x53b091d?, 0x2db550e?})
	/usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc00181fee8 sp=0xc00181fe70 pc=0x2df6b79
github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x8cbb3a0?, 0xc001061f50?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc00181ff88 sp=0xc00181fee8 pc=0x3219ff0
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc00181ffe0 sp=0xc00181ff88 pc=0x322cabb
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00181ffe8 sp=0xc00181ffe0 pc=0x2dbdcc1
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 91
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

goroutine 582 gp=0xc001156fc0 m=nil [sync.WaitGroup.Wait, 39 minutes]:
runtime.gopark(0xc00152faf0?, 0x2d4c874?, 0xe0?, 0x70?, 0x7fc2e6f99840?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000e37a90 sp=0xc000e37a70 pc=0x2db550e
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:466
runtime.semacquire1(0xc000ba0110, 0x0, 0x1, 0x0, 0x19)
	/usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc000e37af8 sp=0xc000e37a90 pc=0x2d9159d
sync.runtime_SemacquireWaitGroup(0xc001e214d0?, 0xc0?)
	/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc000e37b30 sp=0xc000e37af8 pc=0x2db6c2e
sync.(*WaitGroup).Wait(0xc000ba0108)
	/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc000e37b58 sp=0xc000e37b30 pc=0x2dda1a5
golang.org/x/sync/errgroup.(*Group).Wait(0xc000ba0100)
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:56 +0x1e fp=0xc000e37b78 sp=0xc000e37b58 pc=0x3aef95e
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters({0x7fc2e6f48e08, 0xc001061f50}, 0xc00116a780, {0xc000a94e88, 0x11}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:352 +0x8ef fp=0xc000e37e20 sp=0xc000e37b78 pc=0x525d7af
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).cleanupResourceGroup(0xc000b203c0, {0x7fc2e6f48e08, 0xc001061f50}, {0xc000a94e88, 0x11}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:436 +0x294 fp=0xc000e37f28 sp=0xc000e37e20 pc=0x527aed4
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7fc2e6f48e08, 0xc001061f50})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:297 +0x14d fp=0xc000e37fc0 sp=0xc000e37f28 pc=0x5279a0d
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:306 +0x28 fp=0xc000e37fe0 sp=0xc000e37fc0 pc=0x5279888
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000e37fe8 sp=0xc000e37fe0 pc=0x2dbdcc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 581
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:290 +0xc5

rax    0xca
rbx    0x0
rcx    0x2dbfae3
rdx    0x0
rdi    0xc1aa530
rsi    0x80
rbp    0x7ffdfbbb2ca0
rsp    0x7ffdfbbb2c58
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x581c
r13    0x1
r14    0xc1a8440
r15    0x1
rip    0x2dbfae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command

v2/internal.extractBodyFunction.func2({0x8ce31a0?, 0xc000507dd0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e fp=0xc001217f88 sp=0xc001217f58 pc=0x322649e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc001217fe0 sp=0xc001217f88 pc=0x323abbb
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001217fe8 sp=0xc001217fe0 pc=0x2dcbcc1
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 89
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

goroutine 122 gp=0xc0016108c0 m=nil [IO wait]:
runtime.gopark(0x13e45e0?, 0x3000000000?, 0x50?, 0xe7?, 0xb?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0010be720 sp=0xc0010be700 pc=0x2dc350e
runtime.netpollblock(0x2e2dd18?, 0x2d4f066?, 0x0?)
	/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc0010be758 sp=0xc0010be720 pc=0x2d82fb7
internal/poll.runtime_pollWait(0x7f2040062600, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc0010be778 sp=0xc0010be758 pc=0x2dc1da5
internal/poll.(*pollDesc).wait(0xc000711d00?, 0xc0015ee500?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0010be7a0 sp=0xc0010be778 pc=0x2e50a47
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000711d00, {0xc0015ee500, 0x2500, 0x2500})
	/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc0010be838 sp=0xc0010be7a0 pc=0x2e51e79
net.(*netFD).Read(0xc000711d00, {0xc0015ee500?, 0x4?, 0xc000b0c8c0?})
	/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc0010be880 sp=0xc0010be838 pc=0x30175e5
net.(*conn).Read(0xc000f0c380, {0xc0015ee500?, 0x7f203052a2c8?, 0x7f2088ccef30?})
	/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc0010be8c8 sp=0xc0010be880 pc=0x3029685
crypto/tls.(*atLeastReader).Read(0xc0018120f0, {0xc0015ee500?, 0x81d5001?, 0x4?})
	/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc0010be910 sp=0xc0010be8c8 pc=0x309511b
bytes.(*Buffer).ReadFrom(0xc000c189a8, {0x8c2be00, 0xc0018120f0})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc0010be968 sp=0xc0010be910 pc=0x2e8ec78
crypto/tls.(*Conn).readFromUntil(0xc000c18708, {0x8c2cbe0, 0xc000f0c380}, 0xc0010be9d0?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc0010be9a0 sp=0xc0010be968 pc=0x30952fe
crypto/tls.(*Conn).readRecordOrCCS(0xc000c18708, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc0010bec18 sp=0xc0010be9a0 pc=0x309237b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000c18708, {0xc000f8c000, 0x1000, 0x34d7a80?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc0010bec80 sp=0xc0010bec18 pc=0x3098c65
bufio.(*Reader).Read(0xc0014696e0, {0xc0001362e4, 0x9, 0x34e5b2e?})
	/usr/lib/golang/src/bufio/bufio.go:245 +0x197 fp=0xc0010becb8 sp=0xc0010bec80 pc=0x2f2b1d7
io.ReadAtLeast({0x8c2b400, 0xc0014696e0}, {0xc0001362e4, 0x9, 0x9}, 0x9)
	/usr/lib/golang/src/io/io.go:335 +0x8e fp=0xc0010bed00 sp=0xc0010becb8 pc=0x2e1de8e
io.ReadFull(...)
	/usr/lib/golang/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0001362e4, 0x9, 0xc00000016b?}, {0x8c2b400?, 0xc0014696e0?})
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:250 +0x65 fp=0xc0010bed50 sp=0xc0010bed00 pc=0x34ce845
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc0001362a0)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:513 +0x6b fp=0xc0010bedd8 sp=0xc0010bed50 pc=0x34cf40b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0001362a0)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:572 +0x18 fp=0xc0010bee18 sp=0xc0010bedd8 pc=0x34cfb78
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc0010befa8)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2254 +0xca fp=0xc0010bef60 sp=0xc0010bee18 pc=0x34e2a6a
golang.org/x/net/http2.(*ClientConn).readLoop(0xc001680540)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2123 +0x52 fp=0xc0010befc8 sp=0xc0010bef60 pc=0x34e1df2
golang.org/x/net/http2.(*Transport).newClientConn.gowrap1()
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0x25 fp=0xc0010befe0 sp=0xc0010befc8 pc=0x34db585
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0010befe8 sp=0xc0010befe0 pc=0x2dcbcc1
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 121
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0xda5

rax    0xca
rbx    0x0
rcx    0x2dcdae3
rdx    0x0
rdi    0xc1e55d0
rsi    0x80
rbp    0x7fff5673be30
rsp    0x7fff5673bde8
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x0
r13    0x1
r14    0xc1e34e0
r15    0x7
rip    0x2dcdae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
v2/internal.extractBodyFunction.func2({0x8ce31a0?, 0xc000507dd0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e fp=0xc001217f88 sp=0xc001217f58 pc=0x322649e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc001217fe0 sp=0xc001217f88 pc=0x323abbb
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001217fe8 sp=0xc001217fe0 pc=0x2dcbcc1
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 89
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

goroutine 122 gp=0xc0016108c0 m=nil [IO wait]:
runtime.gopark(0x13e45e0?, 0x3000000000?, 0x50?, 0xe7?, 0xb?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0010be720 sp=0xc0010be700 pc=0x2dc350e
runtime.netpollblock(0x2e2dd18?, 0x2d4f066?, 0x0?)
	/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc0010be758 sp=0xc0010be720 pc=0x2d82fb7
internal/poll.runtime_pollWait(0x7f2040062600, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc0010be778 sp=0xc0010be758 pc=0x2dc1da5
internal/poll.(*pollDesc).wait(0xc000711d00?, 0xc0015ee500?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0010be7a0 sp=0xc0010be778 pc=0x2e50a47
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000711d00, {0xc0015ee500, 0x2500, 0x2500})
	/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc0010be838 sp=0xc0010be7a0 pc=0x2e51e79
net.(*netFD).Read(0xc000711d00, {0xc0015ee500?, 0x4?, 0xc000b0c8c0?})
	/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc0010be880 sp=0xc0010be838 pc=0x30175e5
net.(*conn).Read(0xc000f0c380, {0xc0015ee500?, 0x7f203052a2c8?, 0x7f2088ccef30?})
	/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc0010be8c8 sp=0xc0010be880 pc=0x3029685
crypto/tls.(*atLeastReader).Read(0xc0018120f0, {0xc0015ee500?, 0x81d5001?, 0x4?})
	/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc0010be910 sp=0xc0010be8c8 pc=0x309511b
bytes.(*Buffer).ReadFrom(0xc000c189a8, {0x8c2be00, 0xc0018120f0})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc0010be968 sp=0xc0010be910 pc=0x2e8ec78
crypto/tls.(*Conn).readFromUntil(0xc000c18708, {0x8c2cbe0, 0xc000f0c380}, 0xc0010be9d0?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc0010be9a0 sp=0xc0010be968 pc=0x30952fe
crypto/tls.(*Conn).readRecordOrCCS(0xc000c18708, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc0010bec18 sp=0xc0010be9a0 pc=0x309237b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000c18708, {0xc000f8c000, 0x1000, 0x34d7a80?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc0010bec80 sp=0xc0010bec18 pc=0x3098c65
bufio.(*Reader).Read(0xc0014696e0, {0xc0001362e4, 0x9, 0x34e5b2e?})
	/usr/lib/golang/src/bufio/bufio.go:245 +0x197 fp=0xc0010becb8 sp=0xc0010bec80 pc=0x2f2b1d7
io.ReadAtLeast({0x8c2b400, 0xc0014696e0}, {0xc0001362e4, 0x9, 0x9}, 0x9)
	/usr/lib/golang/src/io/io.go:335 +0x8e fp=0xc0010bed00 sp=0xc0010becb8 pc=0x2e1de8e
io.ReadFull(...)
	/usr/lib/golang/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0001362e4, 0x9, 0xc00000016b?}, {0x8c2b400?, 0xc0014696e0?})
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:250 +0x65 fp=0xc0010bed50 sp=0xc0010bed00 pc=0x34ce845
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc0001362a0)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:513 +0x6b fp=0xc0010bedd8 sp=0xc0010bed50 pc=0x34cf40b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0001362a0)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:572 +0x18 fp=0xc0010bee18 sp=0xc0010bedd8 pc=0x34cfb78
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc0010befa8)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2254 +0xca fp=0xc0010bef60 sp=0xc0010bee18 pc=0x34e2a6a
golang.org/x/net/http2.(*ClientConn).readLoop(0xc001680540)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2123 +0x52 fp=0xc0010befc8 sp=0xc0010bef60 pc=0x34e1df2
golang.org/x/net/http2.(*Transport).newClientConn.gowrap1()
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0x25 fp=0xc0010befe0 sp=0xc0010befc8 pc=0x34db585
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0010befe8 sp=0xc0010befe0 pc=0x2dcbcc1
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 121
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0xda5

rax    0xca
rbx    0x0
rcx    0x2dcdae3
rdx    0x0
rdi    0xc1e55d0
rsi    0x80
rbp    0x7fff5673be30
rsp    0x7fff5673bde8
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x0
r13    0x1
r14    0xc1e34e0
r15    0x7
rip    0x2dcdae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command

Contributing tests (6)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Engineering should be able to retrieve kusto logs for a cluster and services	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19	1

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-04-02 17:28	PR #4004	2039756884201705472
2026-04-01 09:27	PR #4693	2039273320041418752
2026-04-01 09:26	PR #4695	2039273285383884800
2026-04-01 09:26	PR #4696	2039273255478497280
2026-03-30 17:50	PR #4688	2038675791503429632

client rate limiter Wait returned an error: context deadline exceeded

full failure pattern:

client rate limiter Wait returned an error: context deadline exceeded

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error:
    <*errors.joinError | 0xc001422a98>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    {
        errs: [
            <context.deadlineExceededError>{},
            <*fmt.wrapError | 0xc000b08900>{
                msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded",
                err: <*fmt.wrapError | 0xc000b088e0>{
                    msg: "client rate limiter Wait returned an error: context deadline exceeded",
                    err: <context.deadlineExceededError>{},
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error:
    <*errors.joinError | 0xc001422a98>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    {
        errs: [
            <context.deadlineExceededError>{},
            <*fmt.wrapError | 0xc000b08900>{
                msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded",
                err: <*fmt.wrapError | 0xc000b088e0>{
                    msg: "client rate limiter Wait returned an error: context deadline exceeded",
                    err: <context.deadlineExceededError>{},
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error:
    <*errors.joinError | 0xc0018416f8>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    {
        errs: [
            <context.deadlineExceededError>{},
            <*fmt.wrapError | 0xc000a10f40>{
                msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded",
                err: <*fmt.wrapError | 0xc000a10ee0>{
                    msg: "client rate limiter Wait returned an error: context deadline exceeded",
                    err: <context.deadlineExceededError>{},
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error:
    <*errors.joinError | 0xc0018416f8>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    {
        errs: [
            <context.deadlineExceededError>{},
            <*fmt.wrapError | 0xc000a10f40>{
                msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded",
                err: <*fmt.wrapError | 0xc000a10ee0>{
                    msg: "client rate limiter Wait returned an error: context deadline exceeded",
                    err: <context.deadlineExceededError>{},
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error:
    <*errors.joinError | 0xc00100cb28>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    {
        errs: [
            <context.deadlineExceededError>{},
            <*fmt.wrapError | 0xc0006dcdc0>{
                msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded",
                err: <*fmt.wrapError | 0xc0006dcda0>{
                    msg: "client rate limiter Wait returned an error: context deadline exceeded",
                    err: <context.deadlineExceededError>{},
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error:
    <*errors.joinError | 0xc00100cb28>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    {
        errs: [
            <context.deadlineExceededError>{},
            <*fmt.wrapError | 0xc0006dcdc0>{
                msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded",
                err: <*fmt.wrapError | 0xc0006dcda0>{
                    msg: "client rate limiter Wait returned an error: context deadline exceeded",
                    err: <context.deadlineExceededError>{},
                },
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	5

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-04-02 16:32	PR #4688	2039742819844755456
2026-04-01 09:26	PR #4702	2039273164030087168
2026-04-01 09:25	PR #4647	2039273018135416832
2026-03-31 20:32	PR #4688	2039078310167711744
2026-03-31 16:24	PR #4688	2039016056621633536

ERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardDdsv6Family...

full failure pattern:

ERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: <url> Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.; provider Microsoft.ContainerService

Signal:

provision

none

Full failure examples (3)

time=2026-03-31T10:19:56.763Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-31T10:19:59.065Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-31T10:20:00.522Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=0742a7f0076c62e490050ef322b8164d2b8ec4a6b2093cc9157dedf9ca7ea606
time=2026-03-31T10:35:03.757Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j9835392-svc/providers/Microsoft.Resources/deployments/0742a7f0076c62e490050ef322b8164d2b8ec4a6b2093cc9157dedf9ca7ea606/operationStatuses/08584266544857542811\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9835392-svc/providers/Microsoft.Resources/deployments/cluster-3eslabsfgrtjg\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9835392-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9835392-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"BadRequest\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9835392-svc/providers/Microsoft.ContainerService/managedClusters/prow-j9835392-svc/agentPools/u64d8ds61\\\",\\r\\n                    \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 8, remaining 6 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n                  },\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"BadRequest\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9835392-svc/providers/Microsoft.ContainerService/managedClusters/prow-j9835392-svc/agentPools/u64d8ds62\\\",\\r\\n                    \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 8, remaining 6 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n                  },\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"BadRequest\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j9835392-svc/providers/Microsoft.ContainerService/managedClusters/prow-j9835392-svc/agentPools/u64d8ds63\\\",\\r\\n                    \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 8, remaining 6 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-03-31T08:56:01.872Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-31T08:56:03.298Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-31T08:56:04.855Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=c2f1fa8aab45f6a7c7e9eb047cbc416616eb154108a2ac9161397547d783bfee
time=2026-03-31T09:10:38.820Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j0598400-svc/providers/Microsoft.Resources/deployments/c2f1fa8aab45f6a7c7e9eb047cbc416616eb154108a2ac9161397547d783bfee/operationStatuses/08584266595214267283\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0598400-svc/providers/Microsoft.Resources/deployments/cluster-hvmosin4uql2c\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0598400-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0598400-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"BadRequest\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0598400-svc/providers/Microsoft.ContainerService/managedClusters/prow-j0598400-svc/agentPools/u64d8ds63\\\",\\r\\n                    \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 8, remaining 6 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n                  },\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"BadRequest\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0598400-svc/providers/Microsoft.ContainerService/managedClusters/prow-j0598400-svc/agentPools/u64d8ds62\\\",\\r\\n                    \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 8, remaining 6 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n                  },\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"BadRequest\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0598400-svc/providers/Microsoft.ContainerService/managedClusters/prow-j0598400-svc/agentPools/u64d8ds61\\\",\\r\\n                    \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 8, remaining 6 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-03-30T21:49:10.674Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-03-30T21:49:12.119Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-03-30T21:49:13.925Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=876d9b3e121f6dd000db312d48993b895f1a6bc589bc9c45390b343351112f98
time=2026-03-30T22:04:17.412Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j0773504-svc/providers/Microsoft.Resources/deployments/876d9b3e121f6dd000db312d48993b895f1a6bc589bc9c45390b343351112f98/operationStatuses/08584266995325433478\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0773504-svc/providers/Microsoft.Resources/deployments/cluster-7tpa2pt5yisy6\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0773504-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0773504-svc/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"BadRequest\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0773504-svc/providers/Microsoft.ContainerService/managedClusters/prow-j0773504-svc/agentPools/u64d8ds63\\\",\\r\\n                    \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 8, remaining 6 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n                  },\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"BadRequest\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0773504-svc/providers/Microsoft.ContainerService/managedClusters/prow-j0773504-svc/agentPools/u64d8ds61\\\",\\r\\n                    \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 8, remaining 6 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n                  },\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"BadRequest\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0773504-svc/providers/Microsoft.ContainerService/managedClusters/prow-j0773504-svc/agentPools/u64d8ds62\\\",\\r\\n                    \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 8, remaining 6 for family StandardDdsv6Family for region westus3. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (1)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster	5

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-03-31 09:54	PR #4364	2038917830379835392
2026-03-31 08:27	PR #4700	2038895896170598400
2026-03-30 21:28	PR #4541	2038730085690773504
2026-03-30 19:03	PR #4507	2038693548202659840
2026-03-30 18:04	PR #4133	2038679684098560000

timeout during CreateHCPClusterAndWait; context deadline exceeded

full failure pattern:

timeout during CreateHCPClusterAndWait; context deadline exceeded

Signal:

e2e

INT, PROD, STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:137]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a32e40>: 
    failed to create HCP cluster np-version-upgrade-cluster-bpwllj, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: failed waiting for cluster="np-version-upgrade-cluster-bpwllj" in resourcegroup="rg-np-version-upgrade-bpwllj-plkrh6" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster np-version-upgrade-cluster-bpwllj, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: failed waiting for cluster=\"np-version-upgrade-cluster-bpwllj\" in resourcegroup=\"rg-np-version-upgrade-bpwllj-plkrh6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009c19a0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6",
            },
            <*fmt.wrapErrors | 0xc000a328a0>{
                msg: "failed waiting for cluster=\"np-version-upgrade-cluster-bpwllj\" in resourcegroup=\"rg-np-version-upgrade-bpwllj-plkrh6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0009c19a0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:137]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a32e40>: 
    failed to create HCP cluster np-version-upgrade-cluster-bpwllj, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: failed waiting for cluster="np-version-upgrade-cluster-bpwllj" in resourcegroup="rg-np-version-upgrade-bpwllj-plkrh6" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster np-version-upgrade-cluster-bpwllj, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: failed waiting for cluster=\"np-version-upgrade-cluster-bpwllj\" in resourcegroup=\"rg-np-version-upgrade-bpwllj-plkrh6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009c19a0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6",
            },
            <*fmt.wrapErrors | 0xc000a328a0>{
                msg: "failed waiting for cluster=\"np-version-upgrade-cluster-bpwllj\" in resourcegroup=\"rg-np-version-upgrade-bpwllj-plkrh6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0009c19a0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bpwllj in resource group rg-np-version-upgrade-bpwllj-plkrh6",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error:
    <*fmt.wrapErrors | 0xc00070ad20>: 
    failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-2lzf9q, error: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2lzf9q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-2lzf9q, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-2lzf9q, error: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-2lzf9q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-2lzf9q, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0017ec0d0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-2lzf9q",
            },
            <*fmt.wrapErrors | 0xc00070acf0>{
                msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-2lzf9q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-2lzf9q, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0017ec0d0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-2lzf9q",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error:
    <*fmt.wrapErrors | 0xc00070ad20>: 
    failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-2lzf9q, error: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2lzf9q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-2lzf9q, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-2lzf9q, error: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-2lzf9q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-2lzf9q, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0017ec0d0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-2lzf9q",
            },
            <*fmt.wrapErrors | 0xc00070acf0>{
                msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-2lzf9q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-2lzf9q, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0017ec0d0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-2lzf9q",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error:
    <*fmt.wrapErrors | 0xc00100f5c0>: 
    failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wh29bc, error: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-wh29bc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wh29bc, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wh29bc, error: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-wh29bc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wh29bc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000f2fd90>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wh29bc",
            },
            <*fmt.wrapErrors | 0xc00100f590>{
                msg: "failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-wh29bc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wh29bc, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000f2fd90>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wh29bc",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error:
    <*fmt.wrapErrors | 0xc00100f5c0>: 
    failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wh29bc, error: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-wh29bc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wh29bc, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wh29bc, error: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-wh29bc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wh29bc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000f2fd90>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wh29bc",
            },
            <*fmt.wrapErrors | 0xc00100f590>{
                msg: "failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-wh29bc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wh29bc, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000f2fd90>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wh29bc",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

Contributing tests (22)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.19	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update node pool labels and taints	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	SRE should be able to log into a cluster via a breakglass session	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	3
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to update nodepool replicas and autoscaling	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	2
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest	1

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-04-02 22:34	PR #4640	2039833971700797440
2026-04-01 09:26	PR #4695	2039273285383884800
2026-03-30 23:43	PR #4695	2038764111487045632
2026-03-30 17:50	PR #4688	2038675791503429632

ERROR CODE: DeploymentNotFound

full failure pattern:

ERROR CODE: DeploymentNotFound

Signal:

e2e

PROD

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:86]: Unexpected error:
    <*fmt.wrapError | 0xc00080e300>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster-6v7grd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-6v7grd/operationStatuses/08584264903894756834
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: DeploymentNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentNotFound",
        "message": "Deployment 'mi-basic-hcp-cluster-6v7grd' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-6v7grd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-6v7grd/operationStatuses/08584264903894756834\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster-6v7grd' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00080e2e0>{
            msg: "failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-6v7grd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-6v7grd/operationStatuses/08584264903894756834\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster-6v7grd' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*fmt.wrapError | 0xc00080e2c0>{
                msg: "failed waiting for deployment \"mi-basic-hcp-cluster-6v7grd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-6v7grd/operationStatuses/08584264903894756834\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster-6v7grd' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*exported.ResponseError | 0xc000e09d70>{
                    ErrorCode: "DeploymentNotFound",
                    StatusCode: 404,
                    RawResponse: {
                        Status: "404 Not Found",
                        StatusCode: 404,
                        Proto: "HTTP/2.0",
                        ProtoMajor: 2,
                        ProtoMinor: 0,
                        Header: {
                            "X-Ms-Request-Id": [
                                "44b167ab-be32-4a33-9212-0cdd2b323e2c",
                            ],
                            "X-Ms-Correlation-Request-Id": [
                                "44b167ab-be32-4a33-9212-0cdd2b323e2c",
                            ],
                            "X-Cache": ["CONFIG_NOCACHE"],
                            "Date": [
                                "Thu, 02 Apr 2026 07:54:57 GMT",
                            ],
                            "Pragma": ["no-cache"],
                            "Content-Length": ["112"],
                            "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                            "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                            "Expires": ["-1"],
                            "X-Ms-Routing-Request-Id": [
                                "CANADAEAST:20260402T075458Z:44b167ab-be32-4a33-9212-0cdd2b323e2c",
                            ],
                            "Stri...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:86]: Unexpected error:
    <*fmt.wrapError | 0xc00080e300>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster-6v7grd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-6v7grd/operationStatuses/08584264903894756834
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: DeploymentNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentNotFound",
        "message": "Deployment 'mi-basic-hcp-cluster-6v7grd' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-6v7grd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-6v7grd/operationStatuses/08584264903894756834\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster-6v7grd' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00080e2e0>{
            msg: "failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-6v7grd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-6v7grd/operationStatuses/08584264903894756834\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster-6v7grd' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*fmt.wrapError | 0xc00080e2c0>{
                msg: "failed waiting for deployment \"mi-basic-hcp-cluster-6v7grd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-6v7grd/operationStatuses/08584264903894756834\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster-6v7grd' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*exported.ResponseError | 0xc000e09d70>{
                    ErrorCode: "DeploymentNotFound",
                    StatusCode: 404,
                    RawResponse: {
                        Status: "404 Not Found",
                        StatusCode: 404,
                        Proto: "HTTP/2.0",
                        ProtoMajor: 2,
                        ProtoMinor: 0,
                        Header: {
                            "X-Ms-Request-Id": [
                                "44b167ab-be32-4a33-9212-0cdd2b323e2c",
                            ],
                            "X-Ms-Correlation-Request-Id": [
                                "44b167ab-be32-4a33-9212-0cdd2b323e2c",
                            ],
                            "X-Cache": ["CONFIG_NOCACHE"],
                            "Date": [
                                "Thu, 02 Apr 2026 07:54:57 GMT",
                            ],
                            "Pragma": ["no-cache"],
                            "Content-Length": ["112"],
                            "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                            "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                            "Expires": ["-1"],
                            "X-Ms-Routing-Request-Id": [
                                "CANADAEAST:20260402T075458Z:44b167ab-be32-4a33-9212-0cdd2b323e2c",
                            ],
                            "Stri...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:151]: Unexpected error:
    <*fmt.wrapError | 0xc000a185a0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster3-dmnsvj" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-dmnsvj/operationStatuses/08584265441919195653
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: DeploymentNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentNotFound",
        "message": "Deployment 'mi-basic-hcp-cluster3-dmnsvj' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster3-dmnsvj\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-dmnsvj/operationStatuses/08584265441919195653\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster3-dmnsvj' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a18580>{
            msg: "failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster3-dmnsvj\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-dmnsvj/operationStatuses/08584265441919195653\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster3-dmnsvj' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*fmt.wrapError | 0xc000a18560>{
                msg: "failed waiting for deployment \"mi-basic-hcp-cluster3-dmnsvj\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-dmnsvj/operationStatuses/08584265441919195653\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster3-dmnsvj' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*exported.ResponseError | 0xc00146a4e0>{
                    ErrorCode: "DeploymentNotFound",
                    StatusCode: 404,
                    RawResponse: {
                        Status: "404 Not Found",
                        StatusCode: 404,
                        Proto: "HTTP/2.0",
                        ProtoMajor: 2,
                        ProtoMinor: 0,
                        Header: {
                            "Pragma": ["no-cache"],
                            "Content-Length": ["113"],
                            "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                            "Date": [
                                "Wed, 01 Apr 2026 16:58:28 GMT",
                            ],
                            "Cache-Control": ["no-cache"],
                            "Expires": ["-1"],
                            "X-Ms-Correlation-Request-Id": [
                                "5adcedce-84d5-4d01-8103-83c4748d8c4f",
                            ],
                            "X-Cache": ["CONFIG_NOCACHE"],
                            "X-Msedge-Ref": [
                                "Ref A: C9E7A214673A436989455F2A631D8C15 Ref B: CHI261140511031 Ref C: 2026-04-01T16:58:28Z",
                            ],
                            "Content-Type": [
                                "application/json; charset=utf-8",
                            ],
                            "X-Ms-Request-Id": [
 ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:151]: Unexpected error:
    <*fmt.wrapError | 0xc000a185a0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster3-dmnsvj" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-dmnsvj/operationStatuses/08584265441919195653
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: DeploymentNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentNotFound",
        "message": "Deployment 'mi-basic-hcp-cluster3-dmnsvj' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster3-dmnsvj\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-dmnsvj/operationStatuses/08584265441919195653\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster3-dmnsvj' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a18580>{
            msg: "failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster3-dmnsvj\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-dmnsvj/operationStatuses/08584265441919195653\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster3-dmnsvj' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*fmt.wrapError | 0xc000a18560>{
                msg: "failed waiting for deployment \"mi-basic-hcp-cluster3-dmnsvj\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-dmnsvj/operationStatuses/08584265441919195653\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-basic-hcp-cluster3-dmnsvj' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*exported.ResponseError | 0xc00146a4e0>{
                    ErrorCode: "DeploymentNotFound",
                    StatusCode: 404,
                    RawResponse: {
                        Status: "404 Not Found",
                        StatusCode: 404,
                        Proto: "HTTP/2.0",
                        ProtoMajor: 2,
                        ProtoMinor: 0,
                        Header: {
                            "Pragma": ["no-cache"],
                            "Content-Length": ["113"],
                            "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                            "Date": [
                                "Wed, 01 Apr 2026 16:58:28 GMT",
                            ],
                            "Cache-Control": ["no-cache"],
                            "Expires": ["-1"],
                            "X-Ms-Correlation-Request-Id": [
                                "5adcedce-84d5-4d01-8103-83c4748d8c4f",
                            ],
                            "X-Cache": ["CONFIG_NOCACHE"],
                            "X-Msedge-Ref": [
                                "Ref A: C9E7A214673A436989455F2A631D8C15 Ref B: CHI261140511031 Ref C: 2026-04-01T16:58:28Z",
                            ],
                            "Content-Type": [
                                "application/json; charset=utf-8",
                            ],
                            "X-Ms-Request-Id": [
 ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:122]: Unexpected error:
    <*fmt.wrapError | 0xc001388140>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-ktlmkb-j8kf65" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-ktlmkb-j8kf65/operationStatuses/08584265438723055136
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: DeploymentNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentNotFound",
        "message": "Deployment 'mi-np-version-upgrade-cluster-ktlmkb-j8kf65' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-version-upgrade-cluster-ktlmkb-j8kf65\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-ktlmkb-j8kf65/operationStatuses/08584265438723055136\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-np-version-upgrade-cluster-ktlmkb-j8kf65' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc001388120>{
            msg: "failed to create managed identities: failed waiting for deployment \"mi-np-version-upgrade-cluster-ktlmkb-j8kf65\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-ktlmkb-j8kf65/operationStatuses/08584265438723055136\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-np-version-upgrade-cluster-ktlmkb-j8kf65' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*fmt.wrapError | 0xc001388100>{
                msg: "failed waiting for deployment \"mi-np-version-upgrade-cluster-ktlmkb-j8kf65\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-ktlmkb-j8kf65/operationStatuses/08584265438723055136\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-np-version-upgrade-cluster-ktlmkb-j8kf65' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*exported.ResponseError | 0xc0008cc1b0>{
                    ErrorCode: "DeploymentNotFound",
                    StatusCode: 404,
                    RawResponse: {
                        Status: "404 Not Found",
                        StatusCode: 404,
                        Proto: "HTTP/2.0",
                        ProtoMajor: 2,
                        ProtoMinor: 0,
                        Header: {
                            "Content-Length": ["128"],
                            "X-Ms-Request-Id": [
                                "00b81d7f-0434-47b0-a11f-b77de2199539",
                            ],
                            "X-Ms-Correlation-Request-Id": [
                                "00b81d7f-0434-47b0-a11f-b77de2199539",
                            ],
                            "Cache-Control": ["no-cache"],
                            "X-Ms-Failure-Cause": ["gateway"],
                            "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                            "Date": [
                                "Wed, 01 Apr 2026 17:03:36 GMT",
                            ],
                            "Content-Type": [
                                "application/json; charset=utf-8",
                            ],
                            "X-Ms-Routing-Request-Id"...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:122]: Unexpected error:
    <*fmt.wrapError | 0xc001388140>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-ktlmkb-j8kf65" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-ktlmkb-j8kf65/operationStatuses/08584265438723055136
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: DeploymentNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentNotFound",
        "message": "Deployment 'mi-np-version-upgrade-cluster-ktlmkb-j8kf65' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-version-upgrade-cluster-ktlmkb-j8kf65\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-ktlmkb-j8kf65/operationStatuses/08584265438723055136\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-np-version-upgrade-cluster-ktlmkb-j8kf65' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc001388120>{
            msg: "failed to create managed identities: failed waiting for deployment \"mi-np-version-upgrade-cluster-ktlmkb-j8kf65\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-ktlmkb-j8kf65/operationStatuses/08584265438723055136\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-np-version-upgrade-cluster-ktlmkb-j8kf65' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*fmt.wrapError | 0xc001388100>{
                msg: "failed waiting for deployment \"mi-np-version-upgrade-cluster-ktlmkb-j8kf65\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-ktlmkb-j8kf65/operationStatuses/08584265438723055136\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-np-version-upgrade-cluster-ktlmkb-j8kf65' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*exported.ResponseError | 0xc0008cc1b0>{
                    ErrorCode: "DeploymentNotFound",
                    StatusCode: 404,
                    RawResponse: {
                        Status: "404 Not Found",
                        StatusCode: 404,
                        Proto: "HTTP/2.0",
                        ProtoMajor: 2,
                        ProtoMinor: 0,
                        Header: {
                            "Content-Length": ["128"],
                            "X-Ms-Request-Id": [
                                "00b81d7f-0434-47b0-a11f-b77de2199539",
                            ],
                            "X-Ms-Correlation-Request-Id": [
                                "00b81d7f-0434-47b0-a11f-b77de2199539",
                            ],
                            "Cache-Control": ["no-cache"],
                            "X-Ms-Failure-Cause": ["gateway"],
                            "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                            "Date": [
                                "Wed, 01 Apr 2026 17:03:36 GMT",
                            ],
                            "Content-Type": [
                                "application/json; charset=utf-8",
                            ],
                            "X-Ms-Routing-Request-Id"...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Contributing tests (8)

Failed At	Job	Test	Support
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.19	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should be able to list HCP clusters without node pools at the resource group level	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-04-02 07:11	PR #4705	2039601679703216128
2026-04-01 16:03	PR #4721	2039373131814014976
2026-04-01 15:57	PR #4693	2039371628005036032
2026-03-30 08:51	PR #4671	2038539478926102528

ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created.

full failure pattern:

ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created.

Signal:

provision

none

Full failure examples (3)

time=2026-04-02T15:45:11.707Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-04-02T15:45:13.158Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-04-02T15:45:15.083Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=0954c7b8221d8b802b1789f3b78b55cab36ed52110ea91016298121b353b80b4
time=2026-04-02T16:00:18.519Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j4296192-svc/providers/Microsoft.Resources/deployments/0954c7b8221d8b802b1789f3b78b55cab36ed52110ea91016298121b353b80b4/operationStatuses/08584264621714434367\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4296192-svc/providers/Microsoft.Resources/deployments/metrics-infra\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n            \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-04-02T15:46:48.325Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-04-02T15:46:50.935Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-04-02T15:46:52.408Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=11f5679ed3673e4a9929371cce32c945149645404637979cf67d14076078d9bb
time=2026-04-02T16:02:26.220Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j4376448-svc/providers/Microsoft.Resources/deployments/11f5679ed3673e4a9929371cce32c945149645404637979cf67d14076078d9bb/operationStatuses/08584264620738937229\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/aks-svc-kv-access-frontend-cert-prow-j4376448\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n            \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/aks-svc-kv-access-admin-api-cert-prow-j4376448\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n            \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/aksSPCRead-sessiongate-cert-prow-j4376448\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n            \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4376448-svc/providers/Microsoft.Resources/deployments/metrics-infra\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n            \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-04-02T15:47:34.702Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-04-02T15:47:35.558Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster
time=2026-04-02T15:47:36.998Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster deployment=a42c05c78c4bc79396dff56a02b6117a0992c06d730dc45d2f0f97c73701fdea
time=2026-04-02T15:59:09.654Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8920960-mgmt-1/providers/Microsoft.Resources/deployments/a42c05c78c4bc79396dff56a02b6117a0992c06d730dc45d2f0f97c73701fdea/operationStatuses/08584264620293070923\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8920960-mgmt-1/providers/Microsoft.Resources/deployments/cluster-4f5djyc2g2fjc\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/1ddf10d9-e1b0-5dfe-8627-4a983a2a3ab1\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/1ddf10d9-e1b0-5dfe-8627-4a983a2a3ab1\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n                    \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          },\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/d4072e7f-2678-5f20-830e-54226531e4b5\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/d4072e7f-2678-5f20-830e-54226531e4b5\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n                    \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          },\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/388c60cd-66c0-5a3c-bf90-1d1bb63fb700\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/388c60cd-66c0-5a3c-bf90-1d1bb63fb700\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n                    \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          },\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/2c104537-ef93-5756-94dd-a4f0b7a92d1f\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/2c104537-ef93-5756-94dd-a4f0b7a92d1f\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n                    \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

Contributing tests (2)

Failed At	Job	Test	Support
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster	3
provision	pull-ci-Azure-ARO-HCP-main-e2e-parallel	Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster	1

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-04-02 15:21	PR #4651	2039724880844296192
2026-04-02 15:21	PR #3960	2039724865174376448
2026-04-02 15:21	PR #4601	2039724856378920960
2026-04-01 20:02	PR #4507	2039433353752481792

Environment: INT

E2E Jobs

Success Rate

70.83%

E2E success

70.83% (17/24)

Daily Run Outcomes

Successful runs Provision failures E2E failures Other failures

6 runs

3 runs

3 runs

6 runs

2 runs

3 runs

1 run

Pass rate	Runs	Date (UTC)	Suite	Test
86.36%	22	2026-04-07	integration/parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API
86.67%	15	2026-04-07	integration/parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19
90.91%	22	2026-04-07	integration/parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors
90.91%	22	2026-04-07	integration/parallel	Customer should be able to create an HCP cluster with back-level version 4.19
90.91%	22	2026-04-07	integration/parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully

Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.

Jump to the full Failure Patterns view for this window

Failure pattern

Failed at

Trend

Also in

timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds

full failure pattern:

timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds

Signal:

e2e

PROD, STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
    <*fmt.wrapErrors | 0xc0016e4ea0>: 
    failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-89jv5j" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-89jv5j, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-89jv5j\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-89jv5j, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000875680>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-89jv5j",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
    <*fmt.wrapErrors | 0xc0016e4ea0>: 
    failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-89jv5j" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-89jv5j, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-89jv5j\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-89jv5j, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000875680>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-89jv5j",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc000d5a900>: 
    failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-ptsj98" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-ptsj98, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-ptsj98\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-ptsj98, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0006ffb60>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-ptsj98",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc000d5a900>: 
    failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-ptsj98" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-ptsj98, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-ptsj98\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-ptsj98, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0006ffb60>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-ptsj98",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:106]: Unexpected error:
    <*fmt.wrapErrors | 0xc00110a3f0>: 
    failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-fkjbdc" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-fkjbdc, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-fkjbdc\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-fkjbdc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e074c0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-fkjbdc",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:106]: Unexpected error:
    <*fmt.wrapErrors | 0xc00110a3f0>: 
    failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-fkjbdc" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-fkjbdc, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-fkjbdc\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-fkjbdc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e074c0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-fkjbdc",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallel	Customer should be able to create an HCP cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-03-30 15:03	n/a	2038633291069788160

found 1 managed resource groups left behind HCP clusters in cluster-listing-<id>

full failure pattern:

found 1 managed resource groups left behind HCP clusters in cluster-listing-<id>

Signal:

e2e

PROD

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246]: Unexpected error:
    <*errors.joinError | 0xc000e4a0f0>: 
    found 1 managed resource groups left behind HCP clusters in cluster-listing-xk5f98
    {
        errs: [
            <*errors.errorString | 0xc00198c880>{
                s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-xk5f98",
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246]: Unexpected error:
    <*errors.joinError | 0xc000e4a0f0>: 
    found 1 managed resource groups left behind HCP clusters in cluster-listing-xk5f98
    {
        errs: [
            <*errors.errorString | 0xc00198c880>{
                s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-xk5f98",
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallel	Customer should be able to list HCP clusters without node pools at the resource group level	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-03-31 08:04	n/a	2038890153002405888

CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded

full failure pattern:

CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded

Signal:

e2e

DEV

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc000b37470>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-s2z7xg, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-s2z7xg" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-s2z7xg, error: failed waiting for nodepool=\"np-1\" for cluster \"private-kv-cluster\" in resourcegroup=\"private-keyvault-s2z7xg\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000340210>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-s2z7xg",
            },
            <*fmt.wrapError | 0xc000f90d40>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"private-kv-cluster\" in resourcegroup=\"private-keyvault-s2z7xg\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc000b37470>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-s2z7xg, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-s2z7xg" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-s2z7xg, error: failed waiting for nodepool=\"np-1\" for cluster \"private-kv-cluster\" in resourcegroup=\"private-keyvault-s2z7xg\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000340210>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-s2z7xg",
            },
            <*fmt.wrapError | 0xc000f90d40>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"private-kv-cluster\" in resourcegroup=\"private-keyvault-s2z7xg\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-03 19:38	n/a	2040151865483595776

add password: error status code received from the API

full failure pattern:

add password: error status code received from the API

Signal:

e2e

DEV, PROD, STG

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:252]: Unexpected error:
    <*fmt.wrapError | 0xc0015d1e60>: 
    add password: error status code received from the API
    {
        msg: "add password: error status code received from the API",
        err: <*odataerrors.ODataError | 0xc0001d6e40>{
            ApiError: {
                Message: "",
                ResponseStatusCode: 404,
                ResponseHeaders: {
                    header: {
                        headers: {
                            "x-ms-ags-diagnostic": {
                                "{\"ServerInfo\":{\"DataCenter\":\"Canada Central\",\"Slice\":\"E\",\"Ring\":\"3\",\"ScaleUnit\":\"000\",\"RoleInstance\":\"TO1PEPF0000542E\"}}": {},
                            },
                            "x-ms-resource-unit": {"1": {}},
                            "date": {
                                "Tue, 31 Mar 2026 08:28:07 GMT": {},
                            },
                            "cache-control": {"no-cache": {}},
                            "strict-transport-security": {
                                "max-age=31536000": {},
                            },
                            "client-request-id": {
                                "75a0d347-72fb-4ab2-924d-ef1861bd1c25": {},
                            },
                            "content-type": {
                                "application/json": {},
                            },
                            "vary": {
                                "Accept-Encoding": {},
                            },
                            "request-id": {
                                "75a0d347-72fb-4ab2-924d-ef1861bd1c25": {},
                            },
                        },
                    },
                },
            },
            backingStore: <*store.InMemoryBackingStore | 0xc0015d1e00>{
                returnOnlyChangedValues: false,
                initializationCompleted: false,
                store: {
                    "errorEscaped": <*odataerrors.MainError | 0xc00070ae40>{
                        backingStore: <*store.InMemoryBackingStore | 0xc0015d1e20>{
                            returnOnlyChangedValues: false,
                            initializationCompleted: false,
                            store: {
                                "code": <*string | 0xc00070ac90>"Request_ResourceNotFound",
                                "message": <*string | 0xc00070acf0>"Resource '3ee33ce9-ce09-4583-b01c-acdbc683ee4c' does not exist or one of its queried reference-property objects are not present.",
                                "innerError": <*odataerrors.InnerError | 0xc00070aea0>{
                                    backingStore: <*store.InMemoryBackingStore | 0xc0015d1e40>{
                                        returnOnlyChangedValues: false,
                                        initializationCompleted: false,
                                        store: {...: ..., ...: ...},
                                        subscribers: {},
                                        changedValues: {...: ..., ...: ...},
                                    },
                                },
                            },
                            subscribers: {},
                            changedValues: {"code": false, "message": false, "innerError": false},
                        },
                    },
                },
                subscribers: {},
                changedValues: {
                    "errorEscaped": false,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:252]: Unexpected error:
    <*fmt.wrapError | 0xc0015d1e60>: 
    add password: error status code received from the API
    {
        msg: "add password: error status code received from the API",
        err: <*odataerrors.ODataError | 0xc0001d6e40>{
            ApiError: {
                Message: "",
                ResponseStatusCode: 404,
                ResponseHeaders: {
                    header: {
                        headers: {
                            "x-ms-ags-diagnostic": {
                                "{\"ServerInfo\":{\"DataCenter\":\"Canada Central\",\"Slice\":\"E\",\"Ring\":\"3\",\"ScaleUnit\":\"000\",\"RoleInstance\":\"TO1PEPF0000542E\"}}": {},
                            },
                            "x-ms-resource-unit": {"1": {}},
                            "date": {
                                "Tue, 31 Mar 2026 08:28:07 GMT": {},
                            },
                            "cache-control": {"no-cache": {}},
                            "strict-transport-security": {
                                "max-age=31536000": {},
                            },
                            "client-request-id": {
                                "75a0d347-72fb-4ab2-924d-ef1861bd1c25": {},
                            },
                            "content-type": {
                                "application/json": {},
                            },
                            "vary": {
                                "Accept-Encoding": {},
                            },
                            "request-id": {
                                "75a0d347-72fb-4ab2-924d-ef1861bd1c25": {},
                            },
                        },
                    },
                },
            },
            backingStore: <*store.InMemoryBackingStore | 0xc0015d1e00>{
                returnOnlyChangedValues: false,
                initializationCompleted: false,
                store: {
                    "errorEscaped": <*odataerrors.MainError | 0xc00070ae40>{
                        backingStore: <*store.InMemoryBackingStore | 0xc0015d1e20>{
                            returnOnlyChangedValues: false,
                            initializationCompleted: false,
                            store: {
                                "code": <*string | 0xc00070ac90>"Request_ResourceNotFound",
                                "message": <*string | 0xc00070acf0>"Resource '3ee33ce9-ce09-4583-b01c-acdbc683ee4c' does not exist or one of its queried reference-property objects are not present.",
                                "innerError": <*odataerrors.InnerError | 0xc00070aea0>{
                                    backingStore: <*store.InMemoryBackingStore | 0xc0015d1e40>{
                                        returnOnlyChangedValues: false,
                                        initializationCompleted: false,
                                        store: {...: ..., ...: ...},
                                        subscribers: {},
                                        changedValues: {...: ..., ...: ...},
                                    },
                                },
                            },
                            subscribers: {},
                            changedValues: {"code": false, "message": false, "innerError": false},
                        },
                    },
                },
                subscribers: {},
                changedValues: {
                    "errorEscaped": false,
                },
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-03-31 08:04	n/a	2038890153002405888

failed to cleanup resource group: failed waiting for resourcegroup="<resource-group>" to finish deleting: context canceled

full failure pattern:

failed to cleanup resource group: failed waiting for resourcegroup="<resource-group>" to finish deleting: context canceled

Signal:

e2e

DEV

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001238348>: 
    failed to cleanup resource group: failed waiting for resourcegroup="rg-negative-tests-pgqhpg" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001245340>{
                msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"rg-negative-tests-pgqhpg\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001245320>{
                    msg: "failed waiting for resourcegroup=\"rg-negative-tests-pgqhpg\" to finish deleting: context canceled",
                    err: <*errors.errorString | 0xbee2c60>{
                        s: "context canceled",
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001238348>: 
    failed to cleanup resource group: failed waiting for resourcegroup="rg-negative-tests-pgqhpg" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001245340>{
                msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"rg-negative-tests-pgqhpg\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001245320>{
                    msg: "failed waiting for resourcegroup=\"rg-negative-tests-pgqhpg\" to finish deleting: context canceled",
                    err: <*errors.errorString | 0xbee2c60>{
                        s: "context canceled",
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-03 19:38	n/a	2040151865483595776

failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded

full failure pattern:

failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded

Signal:

e2e

DEV, PROD

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:186]: Unexpected error:
    <*fmt.wrapError | 0xc001b03140>: 
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="e2e-cilium-96sc9p" to finish creating: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"cilium-np\" for cluster \"cilium-cluster\" in resourcegroup=\"e2e-cilium-96sc9p\" to finish creating: context deadline exceeded",
        err: <context.deadlineExceededError>{},
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:186]: Unexpected error:
    <*fmt.wrapError | 0xc001b03140>: 
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="e2e-cilium-96sc9p" to finish creating: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"cilium-np\" for cluster \"cilium-cluster\" in resourcegroup=\"e2e-cilium-96sc9p\" to finish creating: context deadline exceeded",
        err: <context.deadlineExceededError>{},
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-03 22:15	n/a	2040191555695808512

pod "<pod>" is forbidden: error fetching namespace "<namespace>": unable to find annotation openshift.io/sa.scc.uid-range

full failure pattern:

pod "<pod>" is forbidden: error fetching namespace "<namespace>": unable to find annotation openshift.io/sa.scc.uid-range

Signal:

e2e

DEV

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:344]: Unexpected error:
    <*errors.StatusError | 0xc000cf8500>: 
    pods "wi-test-pod" is forbidden: error fetching namespace "e2e-oidc-wi": unable to find annotation openshift.io/sa.scc.uid-range
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "pods \"wi-test-pod\" is forbidden: error fetching namespace \"e2e-oidc-wi\": unable to find annotation openshift.io/sa.scc.uid-range",
            Reason: "Forbidden",
            Details: {Name: "wi-test-pod", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
            Code: 403,
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:344]: Unexpected error:
    <*errors.StatusError | 0xc000cf8500>: 
    pods "wi-test-pod" is forbidden: error fetching namespace "e2e-oidc-wi": unable to find annotation openshift.io/sa.scc.uid-range
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "pods \"wi-test-pod\" is forbidden: error fetching namespace \"e2e-oidc-wi\": unable to find annotation openshift.io/sa.scc.uid-range",
            Reason: "Forbidden",
            Details: {Name: "wi-test-pod", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
            Code: 403,
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-01 23:58	n/a	2039492594290921472

timeout during CreateHCPClusterAndWait; context deadline exceeded

full failure pattern:

timeout during CreateHCPClusterAndWait; context deadline exceeded

Signal:

e2e

DEV, PROD, STG

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:108]: Unexpected error:
    <*fmt.wrapErrors | 0xc000af9800>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wzrl6k" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-wzrl6k, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-wzrl6k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-wzrl6k, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010f08c0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-wzrl6k",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:108]: Unexpected error:
    <*fmt.wrapErrors | 0xc000af9800>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wzrl6k" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-wzrl6k, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-wzrl6k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-wzrl6k, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010f08c0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-wzrl6k",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-04 14:10	n/a	2040431814182440960

x509: certificate signed by unknown authority

full failure pattern:

x509: certificate signed by unknown authority

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:153]: Timed out after 240.002s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    {
        Cert: {
            Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 116, 254, 18, 151, 126, 82, 115, 250, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 48, 52, 48, 50, 49, 53, 52, 54, 50, 52, 90, 23, 13, 50, 55, 48, 52, 48, 50, 49, 53, 52, 54, 50, 52, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 0, 37, 38, 86, 192, 193, 157, 184, 31, 176, 119, 114, 247, 43, 47, 94, 198, 52, 160, 160, 184, 52, 220, 156, 249, 49, 6, 149, 103, 89, 12, 153, 202, 194, 226, 206, 140, 227, 220, 30, 165, 107, 252, 76, 188, 166, 122, 48, 174, 25, 118, 134, 226, 168, 124, 225, 129, 180, 222, 62, 98, 247, 98, 33, 39, 180, 110, 9, 14, 160, 25, 16, 52, 229, 241, 148, 107, 55, 51, 135, 102, 189, 14, 88, 132, 166, 32, 136, 154, 191, 51, 12, 165, 141, 19, 53, 89, 204, 70, 75, 128, 213, 34, 74, 165, 149, 97, 40, 211, 16, 86, 211, 78, 232, 131, 170, 196, 216, 155, 165, 162, 28, 31, 169, 244, 137, 8, 235, 107, 83, 115, 232, 42, 99, 47, 244, 13, 185, 101, 183, 53, 168, 150, 86, 97, 9, 93, 4, 170, 139, 11, 139, 8, 8, 47, 178, 98, 197, 57, 122, 132, 194, 19, 22, 246, 96, 39, 189, 30, 242, 76, 104, 176, 99, 138, 52, 61, 25, 116, 122, 199, 58, 41, 26, 100, 122, 190, 203, 169, 140, 156, 84, 225, 160, 59, 64, 34, 217, 40, 136, 164, 147, 242, 54, 24, 122, 193, 161, 36, 229, 186, 235, 250, 133, 86, 184, 84, 15, 164, 44, 45, 47, 185, 2, 224, 48, 152, 254, 208, 237, 153, 193, 94, 125, 60, 135, 190, 231, 145, 249, 127, 59, 28, 200, 167, 244, 37, 79, 159, 252, 220, 83, 119, 14, 187, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 173, 7, 154, 228, 95, 210, 182, 32, 107, 72, 222, 221, 69, 250, 120, 233, 184, 185, 183, 13, 130, 51, 206, 63, 239, 225, 132, 158, 79, 79, 157, 133, 252, 4, 91, 30, 68, 28, 172, 150, 204, 130, 31, 71, 250, 97, 135, 180, 168, 190, 30, 3, 155, 158, 26, 161, 100, 236, 60, 119, 139, 196, 154, 240, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 133, 72, 202, 24, 216, 164, 4, 109, 4, 36, 42, 192, 96, 47, 106, 179, 140, 195, 17, 88, 56, 29, 222, 108, 42, 192, 236, 112, 88, 41, 93, 189, 134, 255, 219, 174, 148, 254, 119, 121, 2, 165, 216, 73, 52, 195, 22, 201, 64, 175, 52, 152, 55, 55, 187, 87, 135, 99, 54, 90, 205, 123, 77, 112, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 112, 54, 121, 48, 113, 56, 112, 57, 107, 49, 113, 54, 110, 56, 103, 46, 100, 52, 56, 55, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 75, 184, 166, 203, 165, 66, 231, 128, 240, 212, 230, 170, 132, 69, 115, 243, 121, 10, 100, 191, 1, 215, 138, 96, 8, 49, 116, 82, 205, 128, 16, 67, 90, 42, 120, 221, 78, 220, 68, 41, 219, 33, 156, 169, 205, 200, 56, 150, 110, 198, 166, 99, 43, 64, 200, 119, 89, 214, 36, 97, 17, 104, 82, 73, 255, 234, 140, 195, 184, 245, 66, 52, 226, 61, 181, 232, 25, 74, 115, 83, 130, 189, 5, 144, 8, 104, 50, 44, 208, 97, 118, 148, 50, 225, 49, 104, 109, 191, 218, 198, 232, 195, 183, 241, 159, 48, 11, 52, 194, 197, 6, 115, 43, 0, 49, 15, 55, 5, 166, 236, 105, 150, 42, 123, 77, 102, 132, 182, 200, 134, 141, 64, 47, 30, 163, 77, 129, 164, 58, 252, 129, 78, 194, 225, 2, 29, ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:153]: Timed out after 240.002s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    {
        Cert: {
            Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 116, 254, 18, 151, 126, 82, 115, 250, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 48, 52, 48, 50, 49, 53, 52, 54, 50, 52, 90, 23, 13, 50, 55, 48, 52, 48, 50, 49, 53, 52, 54, 50, 52, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 0, 37, 38, 86, 192, 193, 157, 184, 31, 176, 119, 114, 247, 43, 47, 94, 198, 52, 160, 160, 184, 52, 220, 156, 249, 49, 6, 149, 103, 89, 12, 153, 202, 194, 226, 206, 140, 227, 220, 30, 165, 107, 252, 76, 188, 166, 122, 48, 174, 25, 118, 134, 226, 168, 124, 225, 129, 180, 222, 62, 98, 247, 98, 33, 39, 180, 110, 9, 14, 160, 25, 16, 52, 229, 241, 148, 107, 55, 51, 135, 102, 189, 14, 88, 132, 166, 32, 136, 154, 191, 51, 12, 165, 141, 19, 53, 89, 204, 70, 75, 128, 213, 34, 74, 165, 149, 97, 40, 211, 16, 86, 211, 78, 232, 131, 170, 196, 216, 155, 165, 162, 28, 31, 169, 244, 137, 8, 235, 107, 83, 115, 232, 42, 99, 47, 244, 13, 185, 101, 183, 53, 168, 150, 86, 97, 9, 93, 4, 170, 139, 11, 139, 8, 8, 47, 178, 98, 197, 57, 122, 132, 194, 19, 22, 246, 96, 39, 189, 30, 242, 76, 104, 176, 99, 138, 52, 61, 25, 116, 122, 199, 58, 41, 26, 100, 122, 190, 203, 169, 140, 156, 84, 225, 160, 59, 64, 34, 217, 40, 136, 164, 147, 242, 54, 24, 122, 193, 161, 36, 229, 186, 235, 250, 133, 86, 184, 84, 15, 164, 44, 45, 47, 185, 2, 224, 48, 152, 254, 208, 237, 153, 193, 94, 125, 60, 135, 190, 231, 145, 249, 127, 59, 28, 200, 167, 244, 37, 79, 159, 252, 220, 83, 119, 14, 187, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 173, 7, 154, 228, 95, 210, 182, 32, 107, 72, 222, 221, 69, 250, 120, 233, 184, 185, 183, 13, 130, 51, 206, 63, 239, 225, 132, 158, 79, 79, 157, 133, 252, 4, 91, 30, 68, 28, 172, 150, 204, 130, 31, 71, 250, 97, 135, 180, 168, 190, 30, 3, 155, 158, 26, 161, 100, 236, 60, 119, 139, 196, 154, 240, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 133, 72, 202, 24, 216, 164, 4, 109, 4, 36, 42, 192, 96, 47, 106, 179, 140, 195, 17, 88, 56, 29, 222, 108, 42, 192, 236, 112, 88, 41, 93, 189, 134, 255, 219, 174, 148, 254, 119, 121, 2, 165, 216, 73, 52, 195, 22, 201, 64, 175, 52, 152, 55, 55, 187, 87, 135, 99, 54, 90, 205, 123, 77, 112, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 112, 54, 121, 48, 113, 56, 112, 57, 107, 49, 113, 54, 110, 56, 103, 46, 100, 52, 56, 55, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 75, 184, 166, 203, 165, 66, 231, 128, 240, 212, 230, 170, 132, 69, 115, 243, 121, 10, 100, 191, 1, 215, 138, 96, 8, 49, 116, 82, 205, 128, 16, 67, 90, 42, 120, 221, 78, 220, 68, 41, 219, 33, 156, 169, 205, 200, 56, 150, 110, 198, 166, 99, 43, 64, 200, 119, 89, 214, 36, 97, 17, 104, 82, 73, 255, 234, 140, 195, 184, 245, 66, 52, 226, 61, 181, 232, 25, 74, 115, 83, 130, 189, 5, 144, 8, 104, 50, 44, 208, 97, 118, 148, 50, 225, 49, 104, 109, 191, 218, 198, 232, 195, 183, 241, 159, 48, 11, 52, 194, 197, 6, 115, 43, 0, 49, 15, 55, 5, 166, 236, 105, 150, 42, 123, 77, 102, 132, 182, 200, 134, 141, 64, 47, 30, 163, 77, 129, 164, 58, 252, 129, 78, 194, 225, 2, 29, ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallel	Customer should create an HCP cluster and validate TLS certificates	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-02 15:40	n/a	2039729764155002880

Environment: STG

E2E Jobs

Success Rate

25.00%

E2E success

25.00% (5/20)

Daily Run Outcomes

Successful runs Provision failures E2E failures Other failures

4 runs

3 runs

2 runs

6 runs

2 runs

2 runs

1 run

Pass rate	Runs	Date (UTC)	Suite	Test
50.00%	12	2026-04-07	stage/parallel	Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest
58.33%	12	2026-04-07	stage/parallel	Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest
58.82%	17	2026-04-07	stage/parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle
64.71%	17	2026-04-07	stage/parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors
64.71%	17	2026-04-07	stage/parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL

Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.

Jump to the full Failure Patterns view for this window

Failure pattern

Failed at

Trend

Also in

timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds

full failure pattern:

timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds

Signal:

e2e

INT, PROD

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:101]: Unexpected error:
    <*fmt.wrapErrors | 0xc0017689c0>: 
    failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-j8897x" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-j8897x, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-j8897x\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-j8897x, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009f37b0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-j8897x",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:101]: Unexpected error:
    <*fmt.wrapErrors | 0xc0017689c0>: 
    failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-j8897x" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-j8897x, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-j8897x\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-j8897x, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009f37b0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-j8897x",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
    <*fmt.wrapErrors | 0xc00168c360>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-gcpcqw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-gcpcqw, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-gcpcqw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-gcpcqw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000707550>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-gcpcqw",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
    <*fmt.wrapErrors | 0xc00168c360>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-gcpcqw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-gcpcqw, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-gcpcqw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-gcpcqw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000707550>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-gcpcqw",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc0001ff650>: 
    failed waiting for hcpCluster="cluster-zstream-4-21-nvk8p8" in resourcegroup="rg-zstream-upgrade-4-21-2rn9fm" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-nvk8p8 in resource group rg-zstream-upgrade-4-21-2rn9fm, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-nvk8p8\" in resourcegroup=\"rg-zstream-upgrade-4-21-2rn9fm\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-nvk8p8 in resource group rg-zstream-upgrade-4-21-2rn9fm, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0006c5000>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-nvk8p8 in resource group rg-zstream-upgrade-4-21-2rn9fm",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc0001ff650>: 
    failed waiting for hcpCluster="cluster-zstream-4-21-nvk8p8" in resourcegroup="rg-zstream-upgrade-4-21-2rn9fm" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-nvk8p8 in resource group rg-zstream-upgrade-4-21-2rn9fm, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-nvk8p8\" in resourcegroup=\"rg-zstream-upgrade-4-21-2rn9fm\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-nvk8p8 in resource group rg-zstream-upgrade-4-21-2rn9fm, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0006c5000>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-nvk8p8 in resource group rg-zstream-upgrade-4-21-2rn9fm",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

Contributing tests (22)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	5
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	5
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.19	5
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	5
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	5
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	4
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to update node pool labels and taints	4
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.20	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19	1

Affected runs (7)

Date (UTC)	Associated PR	Prow job
2026-04-05 02:00	n/a	2040610572591960064
2026-04-04 02:00	n/a	2040248180829851648
2026-04-03 02:00	n/a	2039885780997902336
2026-04-02 23:24	n/a	2039846445451841536
2026-04-02 20:40	n/a	2039805076003885056
2026-04-02 01:37	n/a	2039517455973355520
2026-04-01 02:00	n/a	2039160988686618624

timeout during CreateHCPClusterAndWait; context deadline exceeded

full failure pattern:

timeout during CreateHCPClusterAndWait; context deadline exceeded

Signal:

e2e

DEV, INT, PROD

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc0013345a0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-hmmjkl, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-hmmjkl" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-hmmjkl, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-hmmjkl, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-hmmjkl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-hmmjkl, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010337e0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-hmmjkl",
            },
            <*fmt.wrapErrors | 0xc001334570>{
                msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-hmmjkl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-hmmjkl, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0010337e0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-hmmjkl",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc0013345a0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-hmmjkl, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-hmmjkl" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-hmmjkl, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-hmmjkl, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-hmmjkl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-hmmjkl, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010337e0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-hmmjkl",
            },
            <*fmt.wrapErrors | 0xc001334570>{
                msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-hmmjkl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-hmmjkl, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0010337e0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-hmmjkl",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:135]: Unexpected error:
    <*fmt.wrapErrors | 0xc000916240>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-pgdlrr" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-pgdlrr, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-pgdlrr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-pgdlrr, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010acbc0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-pgdlrr",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:135]: Unexpected error:
    <*fmt.wrapErrors | 0xc000916240>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-pgdlrr" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-pgdlrr, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-pgdlrr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-pgdlrr, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010acbc0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-pgdlrr",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
    <*fmt.wrapErrors | 0xc001750cf0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-qdzg54, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-qdzg54" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-qdzg54, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-qdzg54, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-qdzg54\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-qdzg54, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fe0f60>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-qdzg54",
            },
            <*fmt.wrapErrors | 0xc001750cc0>{
                msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-qdzg54\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-qdzg54, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000fe0f60>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-qdzg54",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
    <*fmt.wrapErrors | 0xc001750cf0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-qdzg54, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-qdzg54" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-qdzg54, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-qdzg54, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-qdzg54\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-qdzg54, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fe0f60>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-qdzg54",
            },
            <*fmt.wrapErrors | 0xc001750cc0>{
                msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-qdzg54\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-qdzg54, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000fe0f60>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-qdzg54",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

Contributing tests (4)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	4
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-04-04 02:00	n/a	2040248180829851648
2026-03-31 13:36	n/a	2038973601599721472
2026-03-31 02:00	n/a	2038798596739960832
2026-03-30 23:48	n/a	2038765373188214784
2026-03-30 20:54	n/a	2038721439078551552

ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift

Signal:

e2e

PROD

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:155]: Unexpected error:
    <*fmt.wrapError | 0xc00089c2a0>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-cwknbt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-cwknbt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc001702360>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Expires": ["-1"],
                    "X-Ms-Request-Id": [
                        "69968efa-5e53-4e34-a908-436feea7199a",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["3908"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Content-Type-Options": ["nosniff"],
                    "Pragma": ["no-cache"],
                    "Content-Length": ["107"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/879d1d49-767f-4337-a84e-8f1c585bec0a",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260405T024147Z:f9b47596-ceaf-4492-94b7-7e7939ee00f1",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "X-Msedge-Ref": [
                        "Ref A: 8142C57A62E143F1BD408CF8DCE6F3B4 Ref B: CHI261140502031 Ref C: 2026-04-05T02:41:43Z",
                    ],
                    "X-Ms-Correlation-Request-Id": [
                        "f9b47596-ceaf-4492-94b7-7e7939ee00f1",
                    ],
                    "Date": [
                        "Sun, 05 Apr 2026 02:41:46 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Content-Type": ["application/json"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc00089c1e0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:155]: Unexpected error:
    <*fmt.wrapError | 0xc00089c2a0>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-cwknbt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-cwknbt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc001702360>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Expires": ["-1"],
                    "X-Ms-Request-Id": [
                        "69968efa-5e53-4e34-a908-436feea7199a",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["3908"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Content-Type-Options": ["nosniff"],
                    "Pragma": ["no-cache"],
                    "Content-Length": ["107"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/879d1d49-767f-4337-a84e-8f1c585bec0a",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260405T024147Z:f9b47596-ceaf-4492-94b7-7e7939ee00f1",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "X-Msedge-Ref": [
                        "Ref A: 8142C57A62E143F1BD408CF8DCE6F3B4 Ref B: CHI261140502031 Ref C: 2026-04-05T02:41:43Z",
                    ],
                    "X-Ms-Correlation-Request-Id": [
                        "f9b47596-ceaf-4492-94b7-7e7939ee00f1",
                    ],
                    "Date": [
                        "Sun, 05 Apr 2026 02:41:46 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Content-Type": ["application/json"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc00089c1e0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:127]: Unexpected error:
    <*fmt.wrapError | 0xc0005ef0e0>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-f9d294/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-f9d294/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc001357950>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Envoy-Upstream-Service-Time": ["166"],
                    "X-Ms-Correlation-Request-Id": [
                        "a8f26788-567b-4640-a811-43c8b7434031",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Msedge-Ref": [
                        "Ref A: 2EC63483A147487995A6328397F09705 Ref B: CHI261140505054 Ref C: 2026-04-04T02:44:04Z",
                    ],
                    "Date": [
                        "Sat, 04 Apr 2026 02:44:04 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Pragma": ["no-cache"],
                    "Content-Length": ["107"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260404T024405Z:a8f26788-567b-4640-a811-43c8b7434031",
                    ],
                    "Content-Type": ["application/json"],
                    "Expires": ["-1"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/52040d45-044c-4511-88b7-9733c92944b6",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Request-Id": [
                        "12919637-123e-4c90-833e-437777c2019f",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0005eefa0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:127]: Unexpected error:
    <*fmt.wrapError | 0xc0005ef0e0>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-f9d294/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-f9d294/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc001357950>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Envoy-Upstream-Service-Time": ["166"],
                    "X-Ms-Correlation-Request-Id": [
                        "a8f26788-567b-4640-a811-43c8b7434031",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Msedge-Ref": [
                        "Ref A: 2EC63483A147487995A6328397F09705 Ref B: CHI261140505054 Ref C: 2026-04-04T02:44:04Z",
                    ],
                    "Date": [
                        "Sat, 04 Apr 2026 02:44:04 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Pragma": ["no-cache"],
                    "Content-Length": ["107"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260404T024405Z:a8f26788-567b-4640-a811-43c8b7434031",
                    ],
                    "Content-Type": ["application/json"],
                    "Expires": ["-1"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/52040d45-044c-4511-88b7-9733c92944b6",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Request-Id": [
                        "12919637-123e-4c90-833e-437777c2019f",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0005eefa0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:164]: Unexpected error:
    <*fmt.wrapError | 0xc000786700>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tcr7xr-w9cl9v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tcr7xr/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tcr7xr-w9cl9v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tcr7xr/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000984f90>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Expires": ["-1"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260403T024529Z:507906a9-3741-4c35-99cf-b4f6b8fe9a4b",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Msedge-Ref": [
                        "Ref A: 4BC4BC7D598E4A38B774049176C0C5B8 Ref B: CHI261140508025 Ref C: 2026-04-03T02:45:28Z",
                    ],
                    "Cache-Control": ["no-cache"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/75948d99-9fdd-49dc-a774-686a518c88f9",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Date": [
                        "Fri, 03 Apr 2026 02:45:28 GMT",
                    ],
                    "Content-Length": ["107"],
                    "X-Ms-Request-Id": [
                        "e9eebcd1-0444-496b-90c5-43c9c0a830e0",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["247"],
                    "X-Ms-Correlation-Request-Id": [
                        "507906a9-3741-4c35-99cf-b4f6b8fe9a4b",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "Pragma": ["no-cache"],
                    "Content-Type": ["application/json"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000786640>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:164]: Unexpected error:
    <*fmt.wrapError | 0xc000786700>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tcr7xr-w9cl9v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tcr7xr/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tcr7xr-w9cl9v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tcr7xr/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000984f90>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Expires": ["-1"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260403T024529Z:507906a9-3741-4c35-99cf-b4f6b8fe9a4b",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Msedge-Ref": [
                        "Ref A: 4BC4BC7D598E4A38B774049176C0C5B8 Ref B: CHI261140508025 Ref C: 2026-04-03T02:45:28Z",
                    ],
                    "Cache-Control": ["no-cache"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/75948d99-9fdd-49dc-a774-686a518c88f9",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Date": [
                        "Fri, 03 Apr 2026 02:45:28 GMT",
                    ],
                    "Content-Length": ["107"],
                    "X-Ms-Request-Id": [
                        "e9eebcd1-0444-496b-90c5-43c9c0a830e0",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["247"],
                    "X-Ms-Correlation-Request-Id": [
                        "507906a9-3741-4c35-99cf-b4f6b8fe9a4b",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "Pragma": ["no-cache"],
                    "Content-Type": ["application/json"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000786640>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Contributing tests (9)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1

Affected runs (4)

Date (UTC)	Associated PR	Prow job
2026-04-05 02:00	n/a	2040610572591960064
2026-04-04 02:00	n/a	2040248180829851648
2026-04-03 02:00	n/a	2039885780997902336
2026-04-02 01:37	n/a	2039517455973355520

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

Signal:

e2e

PROD

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error:
    <*fmt.wrapError | 0xc000732640>: 
    failed to create HCP cluster cluster-zstream-4-20-7mzwzp: failed waiting for cluster="cluster-zstream-4-20-7mzwzp" in resourcegroup="rg-zstream-upgrade-4-20-tq69zx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb",
      "name": "e3ee4618-26eb-473e-867f-a23c3644e2bb",
      "status": "Failed",
      "startTime": "2026-04-02T17:31:57.472030714Z",
      "endTime": "2026-04-02T17:32:55.049606818Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-20-7mzwzp: failed waiting for cluster=\"cluster-zstream-4-20-7mzwzp\" in resourcegroup=\"rg-zstream-upgrade-4-20-tq69zx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb\",\n  \"name\": \"e3ee4618-26eb-473e-867f-a23c3644e2bb\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T17:31:57.472030714Z\",\n  \"endTime\": \"2026-04-02T17:32:55.049606818Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000732600>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-20-7mzwzp\" in resourcegroup=\"rg-zstream-upgrade-4-20-tq69zx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb\",\n  \"name\": \"e3ee4618-26eb-473e-867f-a23c3644e2bb\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T17:31:57.472030714Z\",\n  \"endTime\": \"2026-04-02T17:32:55.049606818Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0014af200>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/81439e1d-c310-4abd-91e5-f37963acb072",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["499"],
                        "Content-Type": ["application/json"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Msedge-Ref": [
                            "Ref A: 41853D22B093487EB33229F443FF6553 Ref B: CHI261140513025 Ref C: 2026-04-02T17:33:00Z",
                   ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error:
    <*fmt.wrapError | 0xc000732640>: 
    failed to create HCP cluster cluster-zstream-4-20-7mzwzp: failed waiting for cluster="cluster-zstream-4-20-7mzwzp" in resourcegroup="rg-zstream-upgrade-4-20-tq69zx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb",
      "name": "e3ee4618-26eb-473e-867f-a23c3644e2bb",
      "status": "Failed",
      "startTime": "2026-04-02T17:31:57.472030714Z",
      "endTime": "2026-04-02T17:32:55.049606818Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-20-7mzwzp: failed waiting for cluster=\"cluster-zstream-4-20-7mzwzp\" in resourcegroup=\"rg-zstream-upgrade-4-20-tq69zx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb\",\n  \"name\": \"e3ee4618-26eb-473e-867f-a23c3644e2bb\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T17:31:57.472030714Z\",\n  \"endTime\": \"2026-04-02T17:32:55.049606818Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000732600>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-20-7mzwzp\" in resourcegroup=\"rg-zstream-upgrade-4-20-tq69zx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e3ee4618-26eb-473e-867f-a23c3644e2bb\",\n  \"name\": \"e3ee4618-26eb-473e-867f-a23c3644e2bb\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T17:31:57.472030714Z\",\n  \"endTime\": \"2026-04-02T17:32:55.049606818Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0014af200>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/81439e1d-c310-4abd-91e5-f37963acb072",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["499"],
                        "Content-Type": ["application/json"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Msedge-Ref": [
                            "Ref A: 41853D22B093487EB33229F443FF6553 Ref B: CHI261140513025 Ref C: 2026-04-02T17:33:00Z",
                   ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error:
    <*fmt.wrapError | 0xc0008d7580>: 
    failed to create HCP cluster cluster-zstream-4-19-456qg5: failed waiting for cluster="cluster-zstream-4-19-456qg5" in resourcegroup="rg-zstream-upgrade-4-19-p762zb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a",
      "name": "78ee839b-27e8-45ab-9bef-ab361680e06a",
      "status": "Failed",
      "startTime": "2026-04-02T17:27:46.835065402Z",
      "endTime": "2026-04-02T17:27:57.932415489Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-456qg5: failed waiting for cluster=\"cluster-zstream-4-19-456qg5\" in resourcegroup=\"rg-zstream-upgrade-4-19-p762zb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a\",\n  \"name\": \"78ee839b-27e8-45ab-9bef-ab361680e06a\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T17:27:46.835065402Z\",\n  \"endTime\": \"2026-04-02T17:27:57.932415489Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0008d7560>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-19-456qg5\" in resourcegroup=\"rg-zstream-upgrade-4-19-p762zb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a\",\n  \"name\": \"78ee839b-27e8-45ab-9bef-ab361680e06a\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T17:27:46.835065402Z\",\n  \"endTime\": \"2026-04-02T17:27:57.932415489Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc001317380>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Pragma": ["no-cache"],
                        "Content-Length": ["499"],
                        "X-Ms-Request-Id": [
                            "f6c45648-41ff-430b-8947-420c54510b28",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260402T172807Z:e6630a99-063f-431a-9926-65553b1b3548",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/26acb1f6-ebb4-4c9f-9e33-8c891414cd26",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Msedge-Ref": [
                            "Ref A: C88F82DD8BF54F1BA57C9FED21B63FDF Ref B: CHI261140514034 Ref C: 2026-04-02T17:28:07Z",
                        ],
                        "Date": [
                            "Thu, 02 Apr 2026 17:28:07 GMT",
                        ],
...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error:
    <*fmt.wrapError | 0xc0008d7580>: 
    failed to create HCP cluster cluster-zstream-4-19-456qg5: failed waiting for cluster="cluster-zstream-4-19-456qg5" in resourcegroup="rg-zstream-upgrade-4-19-p762zb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a",
      "name": "78ee839b-27e8-45ab-9bef-ab361680e06a",
      "status": "Failed",
      "startTime": "2026-04-02T17:27:46.835065402Z",
      "endTime": "2026-04-02T17:27:57.932415489Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-456qg5: failed waiting for cluster=\"cluster-zstream-4-19-456qg5\" in resourcegroup=\"rg-zstream-upgrade-4-19-p762zb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a\",\n  \"name\": \"78ee839b-27e8-45ab-9bef-ab361680e06a\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T17:27:46.835065402Z\",\n  \"endTime\": \"2026-04-02T17:27:57.932415489Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0008d7560>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-19-456qg5\" in resourcegroup=\"rg-zstream-upgrade-4-19-p762zb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78ee839b-27e8-45ab-9bef-ab361680e06a\",\n  \"name\": \"78ee839b-27e8-45ab-9bef-ab361680e06a\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T17:27:46.835065402Z\",\n  \"endTime\": \"2026-04-02T17:27:57.932415489Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc001317380>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Pragma": ["no-cache"],
                        "Content-Length": ["499"],
                        "X-Ms-Request-Id": [
                            "f6c45648-41ff-430b-8947-420c54510b28",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260402T172807Z:e6630a99-063f-431a-9926-65553b1b3548",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/26acb1f6-ebb4-4c9f-9e33-8c891414cd26",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Msedge-Ref": [
                            "Ref A: C88F82DD8BF54F1BA57C9FED21B63FDF Ref B: CHI261140514034 Ref C: 2026-04-02T17:28:07Z",
                        ],
                        "Date": [
                            "Thu, 02 Apr 2026 17:28:07 GMT",
                        ],
...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: Unexpected error:
    <*fmt.wrapError | 0xc0005873e0>: 
    failed to create HCP cluster np-version-upgrade-cluster-qhj6bm: failed waiting for cluster="np-version-upgrade-cluster-qhj6bm" in resourcegroup="rg-np-version-upgrade-qhj6bm-hp2rjl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6",
      "name": "774a88cb-0063-4a09-b407-ee6a7f59a9e6",
      "status": "Failed",
      "startTime": "2026-04-02T17:22:52.081012471Z",
      "endTime": "2026-04-02T17:23:13.48665448Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster np-version-upgrade-cluster-qhj6bm: failed waiting for cluster=\"np-version-upgrade-cluster-qhj6bm\" in resourcegroup=\"rg-np-version-upgrade-qhj6bm-hp2rjl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6\",\n  \"name\": \"774a88cb-0063-4a09-b407-ee6a7f59a9e6\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T17:22:52.081012471Z\",\n  \"endTime\": \"2026-04-02T17:23:13.48665448Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0005873c0>{
            msg: "failed waiting for cluster=\"np-version-upgrade-cluster-qhj6bm\" in resourcegroup=\"rg-np-version-upgrade-qhj6bm-hp2rjl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6\",\n  \"name\": \"774a88cb-0063-4a09-b407-ee6a7f59a9e6\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T17:22:52.081012471Z\",\n  \"endTime\": \"2026-04-02T17:23:13.48665448Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0010284e0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["498"],
                        "X-Ms-Correlation-Request-Id": [
                            "8a74d4be-8a1d-4808-936e-22bae5b4ccd3",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: 37CA783E55BC4DE4BA0F59B834B5F490 Ref B: CHI261140504062 Ref C: 2026-04-02T17:23:22Z",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Envoy-Upstream-Service-Time": ["20"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260402T172322Z:8a74d4be-8a1d-4808-936e-22bae5b4ccd3",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "c674f25c-3bae-4b96-9f1e...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: Unexpected error:
    <*fmt.wrapError | 0xc0005873e0>: 
    failed to create HCP cluster np-version-upgrade-cluster-qhj6bm: failed waiting for cluster="np-version-upgrade-cluster-qhj6bm" in resourcegroup="rg-np-version-upgrade-qhj6bm-hp2rjl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6",
      "name": "774a88cb-0063-4a09-b407-ee6a7f59a9e6",
      "status": "Failed",
      "startTime": "2026-04-02T17:22:52.081012471Z",
      "endTime": "2026-04-02T17:23:13.48665448Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster np-version-upgrade-cluster-qhj6bm: failed waiting for cluster=\"np-version-upgrade-cluster-qhj6bm\" in resourcegroup=\"rg-np-version-upgrade-qhj6bm-hp2rjl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6\",\n  \"name\": \"774a88cb-0063-4a09-b407-ee6a7f59a9e6\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T17:22:52.081012471Z\",\n  \"endTime\": \"2026-04-02T17:23:13.48665448Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0005873c0>{
            msg: "failed waiting for cluster=\"np-version-upgrade-cluster-qhj6bm\" in resourcegroup=\"rg-np-version-upgrade-qhj6bm-hp2rjl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/774a88cb-0063-4a09-b407-ee6a7f59a9e6\",\n  \"name\": \"774a88cb-0063-4a09-b407-ee6a7f59a9e6\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T17:22:52.081012471Z\",\n  \"endTime\": \"2026-04-02T17:23:13.48665448Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0010284e0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["498"],
                        "X-Ms-Correlation-Request-Id": [
                            "8a74d4be-8a1d-4808-936e-22bae5b4ccd3",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: 37CA783E55BC4DE4BA0F59B834B5F490 Ref B: CHI261140504062 Ref C: 2026-04-02T17:23:22Z",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Envoy-Upstream-Service-Time": ["20"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260402T172322Z:8a74d4be-8a1d-4808-936e-22bae5b4ccd3",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "c674f25c-3bae-4b96-9f1e...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Contributing tests (8)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-04-02 17:14	n/a	2039753351280726016
2026-04-02 08:48	n/a	2039626069157351424

ERROR CODE: LockContention; detail message Failed to acquire lock: timed out; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: LockContention; detail message Failed to acquire lock: timed out; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
    <*fmt.wrapError | 0xc0009f2840>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-k2qhj7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: LockContention
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LockContention",
        "message": "Failed to acquire lock: timed out",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-k2qhj7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"LockContention\",\n    \"message\": \"Failed to acquire lock: timed out\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc00159e990>{
            ErrorCode: "LockContention",
            StatusCode: 503,
            RawResponse: {
                Status: "503 Service Unavailable",
                StatusCode: 503,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/1c5d69eb-b218-43c3-89e1-797546221c53",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Date": [
                        "Thu, 02 Apr 2026 02:10:08 GMT",
                    ],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": ["LockContention"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260402T021008Z:afdf4447-88c0-4010-85eb-2250eadd30fe",
                    ],
                    "X-Msedge-Ref": [
                        "Ref A: 1EC287F8926644E099B85CCB3B5E9590 Ref B: CHI261140504042 Ref C: 2026-04-02T02:09:58Z",
                    ],
                    "Content-Length": ["186"],
                    "Expires": ["-1"],
                    "X-Ms-Request-Id": [
                        "2b953333-0bff-468f-ad05-d66baee5ebf5",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["10006"],
                    "X-Ms-Correlation-Request-Id": [
                        "afdf4447-88c0-4010-85eb-2250eadd30fe",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "Cache-Control": ["no-cache"],
                    "Pragma": ["no-cache"],
                    "Retry-After": ["10"],
                    "X-Ms-Failure-Cause": ["service"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0009f2700>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 76, 111, 99, 107, 67, 111, 110, 116, 101, 110, 116, 105, 111, 110, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 97, 99, 113, 117, 105, 114, 101, 32, 108, 111, 99, 107, 58, 32, 116, 105, 109, 101, 100, 32, 111, 117, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 57, 51, 57, 57, 50, 56, 49, 45, 48, 48, 97, 50, 45, 52, 98, 51, 57, 45, 98, 98, 51, 100, 45, 98, 50, 54, 52, 53, 98, 98, 98, 100, 98, 57, 51, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                Conte...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
    <*fmt.wrapError | 0xc0009f2840>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-k2qhj7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: LockContention
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LockContention",
        "message": "Failed to acquire lock: timed out",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-k2qhj7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"LockContention\",\n    \"message\": \"Failed to acquire lock: timed out\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc00159e990>{
            ErrorCode: "LockContention",
            StatusCode: 503,
            RawResponse: {
                Status: "503 Service Unavailable",
                StatusCode: 503,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/1c5d69eb-b218-43c3-89e1-797546221c53",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Date": [
                        "Thu, 02 Apr 2026 02:10:08 GMT",
                    ],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": ["LockContention"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260402T021008Z:afdf4447-88c0-4010-85eb-2250eadd30fe",
                    ],
                    "X-Msedge-Ref": [
                        "Ref A: 1EC287F8926644E099B85CCB3B5E9590 Ref B: CHI261140504042 Ref C: 2026-04-02T02:09:58Z",
                    ],
                    "Content-Length": ["186"],
                    "Expires": ["-1"],
                    "X-Ms-Request-Id": [
                        "2b953333-0bff-468f-ad05-d66baee5ebf5",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["10006"],
                    "X-Ms-Correlation-Request-Id": [
                        "afdf4447-88c0-4010-85eb-2250eadd30fe",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "Cache-Control": ["no-cache"],
                    "Pragma": ["no-cache"],
                    "Retry-After": ["10"],
                    "X-Ms-Failure-Cause": ["service"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0009f2700>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 76, 111, 99, 107, 67, 111, 110, 116, 101, 110, 116, 105, 111, 110, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 97, 99, 113, 117, 105, 114, 101, 32, 108, 111, 99, 107, 58, 32, 116, 105, 109, 101, 100, 32, 111, 117, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 57, 51, 57, 57, 50, 56, 49, 45, 48, 48, 97, 50, 45, 52, 98, 51, 57, 45, 98, 98, 51, 100, 45, 98, 50, 54, 52, 53, 98, 98, 98, 100, 98, 57, 51, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                Conte...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:103]: Unexpected error:
    <*fmt.wrapError | 0xc00142ee20>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-4qqv6m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: LockContention
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LockContention",
        "message": "Failed to acquire lock: timed out",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-4qqv6m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"LockContention\",\n    \"message\": \"Failed to acquire lock: timed out\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc00104d530>{
            ErrorCode: "LockContention",
            StatusCode: 503,
            RawResponse: {
                Status: "503 Service Unavailable",
                StatusCode: 503,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Content-Type-Options": ["nosniff"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/0949565f-2116-4c2d-ac42-f874e16826f1",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260402T021009Z:c60ff00c-0498-471e-83f9-643c3970b3df",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "X-Msedge-Ref": [
                        "Ref A: 84A2B7061A1D427CAC14B9844929E76E Ref B: CHI261140502054 Ref C: 2026-04-02T02:09:58Z",
                    ],
                    "Pragma": ["no-cache"],
                    "Content-Length": ["186"],
                    "Retry-After": ["10"],
                    "X-Ms-Error-Code": ["LockContention"],
                    "X-Ms-Request-Id": [
                        "342b364f-60a1-493c-a980-5f75e6f1c77f",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["10010"],
                    "Date": [
                        "Thu, 02 Apr 2026 02:10:08 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Expires": ["-1"],
                    "X-Ms-Correlation-Request-Id": [
                        "c60ff00c-0498-471e-83f9-643c3970b3df",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc00142ed60>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 76, 111, 99, 107, 67, 111, 110, 116, 101, 110, 116, 105, 111, 110, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 97, 99, 113, 117, 105, 114, 101, 32, 108, 111, 99, 107, 58, 32, 116, 105, 109, 101, 100, 32, 111, 117, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 57, 51, 57, 57, 50, 56, 49, 45, 48, 48, 97, 50, 45, 52, 98, 51, 57, 45, 98, 98, 51, 100, 45, 98, 50, 54, 52, 53, 98, 98, 98, 100, 98, 57, 51, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                Co...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:103]: Unexpected error:
    <*fmt.wrapError | 0xc00142ee20>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-4qqv6m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: LockContention
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LockContention",
        "message": "Failed to acquire lock: timed out",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-4qqv6m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"LockContention\",\n    \"message\": \"Failed to acquire lock: timed out\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc00104d530>{
            ErrorCode: "LockContention",
            StatusCode: 503,
            RawResponse: {
                Status: "503 Service Unavailable",
                StatusCode: 503,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Content-Type-Options": ["nosniff"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/0949565f-2116-4c2d-ac42-f874e16826f1",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260402T021009Z:c60ff00c-0498-471e-83f9-643c3970b3df",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "X-Msedge-Ref": [
                        "Ref A: 84A2B7061A1D427CAC14B9844929E76E Ref B: CHI261140502054 Ref C: 2026-04-02T02:09:58Z",
                    ],
                    "Pragma": ["no-cache"],
                    "Content-Length": ["186"],
                    "Retry-After": ["10"],
                    "X-Ms-Error-Code": ["LockContention"],
                    "X-Ms-Request-Id": [
                        "342b364f-60a1-493c-a980-5f75e6f1c77f",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["10010"],
                    "Date": [
                        "Thu, 02 Apr 2026 02:10:08 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Expires": ["-1"],
                    "X-Ms-Correlation-Request-Id": [
                        "c60ff00c-0498-471e-83f9-643c3970b3df",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc00142ed60>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 76, 111, 99, 107, 67, 111, 110, 116, 101, 110, 116, 105, 111, 110, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 97, 99, 113, 117, 105, 114, 101, 32, 108, 111, 99, 107, 58, 32, 116, 105, 109, 101, 100, 32, 111, 117, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 57, 51, 57, 57, 50, 56, 49, 45, 48, 48, 97, 50, 45, 52, 98, 51, 57, 45, 98, 98, 51, 100, 45, 98, 50, 54, 52, 53, 98, 98, 98, 100, 98, 57, 51, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                Co...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
    <*fmt.wrapError | 0xc000e760c0>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-pkkxdw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: LockContention
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LockContention",
        "message": "Failed to acquire lock: timed out",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-pkkxdw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"LockContention\",\n    \"message\": \"Failed to acquire lock: timed out\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000dce1e0>{
            ErrorCode: "LockContention",
            StatusCode: 503,
            RawResponse: {
                Status: "503 Service Unavailable",
                StatusCode: 503,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Cache-Control": ["no-cache"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "Content-Length": ["186"],
                    "X-Ms-Error-Code": ["LockContention"],
                    "X-Content-Type-Options": ["nosniff"],
                    "Pragma": ["no-cache"],
                    "Retry-After": ["10"],
                    "X-Ms-Correlation-Request-Id": [
                        "37ef49af-09e9-40ee-bfca-efc082a80f64",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260330T021725Z:37ef49af-09e9-40ee-bfca-efc082a80f64",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Content-Type": ["application/json"],
                    "Expires": ["-1"],
                    "X-Ms-Request-Id": [
                        "e0a372c3-fcdc-4ce9-bf08-e2fce9bc2797",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["10011"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/3345b2b1-e23f-4ce2-8dab-587f9642a08d",
                    ],
                    "X-Msedge-Ref": [
                        "Ref A: 93A8B2C021534273A8B574371656B5C5 Ref B: CH1AA2020620025 Ref C: 2026-03-30T02:17:14Z",
                    ],
                    "Date": [
                        "Mon, 30 Mar 2026 02:17:24 GMT",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000e76000>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 76, 111, 99, 107, 67, 111, 110, 116, 101, 110, 116, 105, 111, 110, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 97, 99, 113, 117, 105, 114, 101, 32, 108, 111, 99, 107, 58, 32, 116, 105, 109, 101, 100, 32, 111, 117, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 57, 51, 57, 57, 50, 56, 49, 45, 48, 48, 97, 50, 45, 52, 98, 51, 57, 45, 98, 98, 51, 100, 45, 98, 50, 54, 52, 53, 98, 98, 98, 100, 98, 57, 51, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
          ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
    <*fmt.wrapError | 0xc000e760c0>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-pkkxdw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: LockContention
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LockContention",
        "message": "Failed to acquire lock: timed out",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-pkkxdw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"LockContention\",\n    \"message\": \"Failed to acquire lock: timed out\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000dce1e0>{
            ErrorCode: "LockContention",
            StatusCode: 503,
            RawResponse: {
                Status: "503 Service Unavailable",
                StatusCode: 503,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Cache-Control": ["no-cache"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "Content-Length": ["186"],
                    "X-Ms-Error-Code": ["LockContention"],
                    "X-Content-Type-Options": ["nosniff"],
                    "Pragma": ["no-cache"],
                    "Retry-After": ["10"],
                    "X-Ms-Correlation-Request-Id": [
                        "37ef49af-09e9-40ee-bfca-efc082a80f64",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260330T021725Z:37ef49af-09e9-40ee-bfca-efc082a80f64",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Content-Type": ["application/json"],
                    "Expires": ["-1"],
                    "X-Ms-Request-Id": [
                        "e0a372c3-fcdc-4ce9-bf08-e2fce9bc2797",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["10011"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/3345b2b1-e23f-4ce2-8dab-587f9642a08d",
                    ],
                    "X-Msedge-Ref": [
                        "Ref A: 93A8B2C021534273A8B574371656B5C5 Ref B: CH1AA2020620025 Ref C: 2026-03-30T02:17:14Z",
                    ],
                    "Date": [
                        "Mon, 30 Mar 2026 02:17:24 GMT",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000e76000>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 76, 111, 99, 107, 67, 111, 110, 116, 101, 110, 116, 105, 111, 110, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 97, 99, 113, 117, 105, 114, 101, 32, 108, 111, 99, 107, 58, 32, 116, 105, 109, 101, 100, 32, 111, 117, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 57, 51, 57, 57, 50, 56, 49, 45, 48, 48, 97, 50, 45, 52, 98, 51, 57, 45, 98, 98, 51, 100, 45, 98, 50, 54, 52, 53, 98, 98, 98, 100, 98, 57, 51, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
          ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Contributing tests (7)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.19	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-04-02 01:37	n/a	2039517455973355520
2026-03-30 02:00	n/a	2038436115614011392

add password: error status code received from the API

full failure pattern:

add password: error status code received from the API

Signal:

e2e

DEV, INT, PROD

Full failure examples (2)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:252]: Unexpected error:
    <*fmt.wrapError | 0xc000517600>: 
    add password: error status code received from the API
    {
        msg: "add password: error status code received from the API",
        err: <*odataerrors.ODataError | 0xc000b0ba40>{
            ApiError: {
                Message: "",
                ResponseStatusCode: 404,
                ResponseHeaders: {
                    header: {
                        headers: {
                            "client-request-id": {
                                "0c31e575-0500-4a44-aebc-155ee4c1eea2": {},
                            },
                            "content-type": {
                                "application/json": {},
                            },
                            "x-ms-resource-unit": {"1": {}},
                            "date": {
                                "Tue, 31 Mar 2026 02:30:32 GMT": {},
                            },
                            "strict-transport-security": {
                                "max-age=31536000": {},
                            },
                            "request-id": {
                                "0c31e575-0500-4a44-aebc-155ee4c1eea2": {},
                            },
                            "x-ms-ags-diagnostic": {
                                "{\"ServerInfo\":{\"DataCenter\":\"Canada Central\",\"Slice\":\"E\",\"Ring\":\"3\",\"ScaleUnit\":\"001\",\"RoleInstance\":\"YT1PEPF00003E4B\"}}": {},
                            },
                            "cache-control": {"no-cache": {}},
                            "vary": {
                                "Accept-Encoding": {},
                            },
                        },
                    },
                },
            },
            backingStore: <*store.InMemoryBackingStore | 0xc000517540>{
                returnOnlyChangedValues: false,
                initializationCompleted: false,
                store: {
                    "errorEscaped": <*odataerrors.MainError | 0xc000741cd0>{
                        backingStore: <*store.InMemoryBackingStore | 0xc000517580>{
                            returnOnlyChangedValues: false,
                            initializationCompleted: false,
                            store: {
                                "innerError": <*odataerrors.InnerError | 0xc000741d30>{
                                    backingStore: <*store.InMemoryBackingStore | 0xc0005175c0>{
                                        returnOnlyChangedValues: false,
                                        initializationCompleted: false,
                                        store: {...: ..., ...: ...},
                                        subscribers: {},
                                        changedValues: {...: ..., ...: ...},
                                    },
                                },
                                "code": <*string | 0xc000741a20>"Request_ResourceNotFound",
                                "message": <*string | 0xc000741a70>"Resource '9c86f191-fe0d-40fc-8883-79c3daf62dcd' does not exist or one of its queried reference-property objects are not present.",
                            },
                            subscribers: {},
                            changedValues: {"innerError": false, "code": false, "message": false},
                        },
                    },
                },
                subscribers: {},
                changedValues: {
                    "errorEscaped": false,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:252]: Unexpected error:
    <*fmt.wrapError | 0xc000517600>: 
    add password: error status code received from the API
    {
        msg: "add password: error status code received from the API",
        err: <*odataerrors.ODataError | 0xc000b0ba40>{
            ApiError: {
                Message: "",
                ResponseStatusCode: 404,
                ResponseHeaders: {
                    header: {
                        headers: {
                            "client-request-id": {
                                "0c31e575-0500-4a44-aebc-155ee4c1eea2": {},
                            },
                            "content-type": {
                                "application/json": {},
                            },
                            "x-ms-resource-unit": {"1": {}},
                            "date": {
                                "Tue, 31 Mar 2026 02:30:32 GMT": {},
                            },
                            "strict-transport-security": {
                                "max-age=31536000": {},
                            },
                            "request-id": {
                                "0c31e575-0500-4a44-aebc-155ee4c1eea2": {},
                            },
                            "x-ms-ags-diagnostic": {
                                "{\"ServerInfo\":{\"DataCenter\":\"Canada Central\",\"Slice\":\"E\",\"Ring\":\"3\",\"ScaleUnit\":\"001\",\"RoleInstance\":\"YT1PEPF00003E4B\"}}": {},
                            },
                            "cache-control": {"no-cache": {}},
                            "vary": {
                                "Accept-Encoding": {},
                            },
                        },
                    },
                },
            },
            backingStore: <*store.InMemoryBackingStore | 0xc000517540>{
                returnOnlyChangedValues: false,
                initializationCompleted: false,
                store: {
                    "errorEscaped": <*odataerrors.MainError | 0xc000741cd0>{
                        backingStore: <*store.InMemoryBackingStore | 0xc000517580>{
                            returnOnlyChangedValues: false,
                            initializationCompleted: false,
                            store: {
                                "innerError": <*odataerrors.InnerError | 0xc000741d30>{
                                    backingStore: <*store.InMemoryBackingStore | 0xc0005175c0>{
                                        returnOnlyChangedValues: false,
                                        initializationCompleted: false,
                                        store: {...: ..., ...: ...},
                                        subscribers: {},
                                        changedValues: {...: ..., ...: ...},
                                    },
                                },
                                "code": <*string | 0xc000741a20>"Request_ResourceNotFound",
                                "message": <*string | 0xc000741a70>"Resource '9c86f191-fe0d-40fc-8883-79c3daf62dcd' does not exist or one of its queried reference-property objects are not present.",
                            },
                            subscribers: {},
                            changedValues: {"innerError": false, "code": false, "message": false},
                        },
                    },
                },
                subscribers: {},
                changedValues: {
                    "errorEscaped": false,
                },
            },
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:252]: Unexpected error:
    <*fmt.wrapError | 0xc000e25420>: 
    add password: error status code received from the API
    {
        msg: "add password: error status code received from the API",
        err: <*odataerrors.ODataError | 0xc00063e4e0>{
            ApiError: {
                Message: "",
                ResponseStatusCode: 404,
                ResponseHeaders: {
                    header: {
                        headers: {
                            "x-ms-ags-diagnostic": {
                                "{\"ServerInfo\":{\"DataCenter\":\"East US\",\"Slice\":\"E\",\"Ring\":\"5\",\"ScaleUnit\":\"007\",\"RoleInstance\":\"MN1PEPF00028332\"}}": {},
                            },
                            "x-ms-resource-unit": {"1": {}},
                            "date": {
                                "Tue, 31 Mar 2026 00:21:31 GMT": {},
                            },
                            "vary": {
                                "Accept-Encoding": {},
                            },
                            "request-id": {
                                "061ef776-ae17-49c6-bbac-b0d62d7d9704": {},
                            },
                            "cache-control": {"no-cache": {}},
                            "content-type": {
                                "application/json": {},
                            },
                            "client-request-id": {
                                "061ef776-ae17-49c6-bbac-b0d62d7d9704": {},
                            },
                            "strict-transport-security": {
                                "max-age=31536000": {},
                            },
                        },
                    },
                },
            },
            backingStore: <*store.InMemoryBackingStore | 0xc000e253c0>{
                returnOnlyChangedValues: false,
                initializationCompleted: false,
                store: {
                    "errorEscaped": <*odataerrors.MainError | 0xc000d26ce0>{
                        backingStore: <*store.InMemoryBackingStore | 0xc000e253e0>{
                            returnOnlyChangedValues: false,
                            initializationCompleted: false,
                            store: {
                                "code": <*string | 0xc000d26b60>"Request_ResourceNotFound",
                                "message": <*string | 0xc000d26bb0>"Resource '02691b9b-c5f7-45b0-9cff-7fc2c9805fcd' does not exist or one of its queried reference-property objects are not present.",
                                "innerError": <*odataerrors.InnerError | 0xc000d26d40>{
                                    backingStore: <*store.InMemoryBackingStore | 0xc000e25400>{
                                        returnOnlyChangedValues: false,
                                        initializationCompleted: false,
                                        store: {...: ..., ...: ...},
                                        subscribers: {},
                                        changedValues: {...: ..., ...: ...},
                                    },
                                },
                            },
                            subscribers: {},
                            changedValues: {"code": false, "message": false, "innerError": false},
                        },
                    },
                },
                subscribers: {},
                changedValues: {
                    "errorEscaped": false,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:252]: Unexpected error:
    <*fmt.wrapError | 0xc000e25420>: 
    add password: error status code received from the API
    {
        msg: "add password: error status code received from the API",
        err: <*odataerrors.ODataError | 0xc00063e4e0>{
            ApiError: {
                Message: "",
                ResponseStatusCode: 404,
                ResponseHeaders: {
                    header: {
                        headers: {
                            "x-ms-ags-diagnostic": {
                                "{\"ServerInfo\":{\"DataCenter\":\"East US\",\"Slice\":\"E\",\"Ring\":\"5\",\"ScaleUnit\":\"007\",\"RoleInstance\":\"MN1PEPF00028332\"}}": {},
                            },
                            "x-ms-resource-unit": {"1": {}},
                            "date": {
                                "Tue, 31 Mar 2026 00:21:31 GMT": {},
                            },
                            "vary": {
                                "Accept-Encoding": {},
                            },
                            "request-id": {
                                "061ef776-ae17-49c6-bbac-b0d62d7d9704": {},
                            },
                            "cache-control": {"no-cache": {}},
                            "content-type": {
                                "application/json": {},
                            },
                            "client-request-id": {
                                "061ef776-ae17-49c6-bbac-b0d62d7d9704": {},
                            },
                            "strict-transport-security": {
                                "max-age=31536000": {},
                            },
                        },
                    },
                },
            },
            backingStore: <*store.InMemoryBackingStore | 0xc000e253c0>{
                returnOnlyChangedValues: false,
                initializationCompleted: false,
                store: {
                    "errorEscaped": <*odataerrors.MainError | 0xc000d26ce0>{
                        backingStore: <*store.InMemoryBackingStore | 0xc000e253e0>{
                            returnOnlyChangedValues: false,
                            initializationCompleted: false,
                            store: {
                                "code": <*string | 0xc000d26b60>"Request_ResourceNotFound",
                                "message": <*string | 0xc000d26bb0>"Resource '02691b9b-c5f7-45b0-9cff-7fc2c9805fcd' does not exist or one of its queried reference-property objects are not present.",
                                "innerError": <*odataerrors.InnerError | 0xc000d26d40>{
                                    backingStore: <*store.InMemoryBackingStore | 0xc000e25400>{
                                        returnOnlyChangedValues: false,
                                        initializationCompleted: false,
                                        store: {...: ..., ...: ...},
                                        subscribers: {},
                                        changedValues: {...: ..., ...: ...},
                                    },
                                },
                            },
                            subscribers: {},
                            changedValues: {"code": false, "message": false, "innerError": false},
                        },
                    },
                },
                subscribers: {},
                changedValues: {
                    "errorEscaped": false,
                },
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-03-31 02:00	n/a	2038798596739960832
2026-03-30 23:48	n/a	2038765373188214784

ERROR CODE: ServiceUnavailable; detail message Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installa...

full failure pattern:

ERROR CODE: ServiceUnavailable; detail message Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
    <*fmt.wrapError | 0xc00075c1e0>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-wvg7xj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-wvg7xj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: ServiceUnavailable
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ServiceUnavailable",
        "message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster patch-name-cluster: failed starting cluster creation \"patch-name-cluster\" in resourcegroup=\"patch-name-wvg7xj\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-wvg7xj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"ServiceUnavailable\",\n    \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00075c1c0>{
            msg: "failed starting cluster creation \"patch-name-cluster\" in resourcegroup=\"patch-name-wvg7xj\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-wvg7xj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"ServiceUnavailable\",\n    \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00004d2f0>{
                ErrorCode: "ServiceUnavailable",
                StatusCode: 503,
                RawResponse: {
                    Status: "503 Service Unavailable",
                    StatusCode: 503,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Envoy-Upstream-Service-Time": ["2878"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "X-Ms-Correlation-Request-Id": [
                            "0b9741c0-9777-4781-b4fb-bf1afce41505",
                        ],
                        "Retry-After": ["59"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "Date": [
                            "Thu, 02 Apr 2026 08:57:47 GMT",
                        ],
                        "Content-Length": ["257"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": ["ServiceUnavailable"],
                        "X-Ms-Request-Id": [
                            "87531355-f28f-4c12-bf8b-c7f8a31db6ce",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/7ad18b62-8874-4d58-81d6-01921cd42fc7",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260402T085747Z:0b9741c0-9777-4781-b4fb-bf1afce41505",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Expires": ["-1"],
                        "X-Ms-...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
    <*fmt.wrapError | 0xc00075c1e0>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-wvg7xj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-wvg7xj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: ServiceUnavailable
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ServiceUnavailable",
        "message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster patch-name-cluster: failed starting cluster creation \"patch-name-cluster\" in resourcegroup=\"patch-name-wvg7xj\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-wvg7xj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"ServiceUnavailable\",\n    \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00075c1c0>{
            msg: "failed starting cluster creation \"patch-name-cluster\" in resourcegroup=\"patch-name-wvg7xj\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-wvg7xj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"ServiceUnavailable\",\n    \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00004d2f0>{
                ErrorCode: "ServiceUnavailable",
                StatusCode: 503,
                RawResponse: {
                    Status: "503 Service Unavailable",
                    StatusCode: 503,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Envoy-Upstream-Service-Time": ["2878"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "X-Ms-Correlation-Request-Id": [
                            "0b9741c0-9777-4781-b4fb-bf1afce41505",
                        ],
                        "Retry-After": ["59"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "Date": [
                            "Thu, 02 Apr 2026 08:57:47 GMT",
                        ],
                        "Content-Length": ["257"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": ["ServiceUnavailable"],
                        "X-Ms-Request-Id": [
                            "87531355-f28f-4c12-bf8b-c7f8a31db6ce",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/7ad18b62-8874-4d58-81d6-01921cd42fc7",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260402T085747Z:0b9741c0-9777-4781-b4fb-bf1afce41505",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Expires": ["-1"],
                        "X-Ms-...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:123]: Unexpected error:
    <*fmt.wrapError | 0xc0004e6700>: 
    failed starting cluster creation "basic-hcp-cluster2" in resourcegroup="customer-rg-p6dr85": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-p6dr85/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster2
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: ServiceUnavailable
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ServiceUnavailable",
        "message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed starting cluster creation \"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-p6dr85\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-p6dr85/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster2\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"ServiceUnavailable\",\n    \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc001402630>{
            ErrorCode: "ServiceUnavailable",
            StatusCode: 503,
            RawResponse: {
                Status: "503 Service Unavailable",
                StatusCode: 503,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Cache-Control": ["no-cache"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Request-Id": [
                        "981e3b54-276e-45f0-a48f-b05f015c79dc",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["8461"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/d750dcff-9cd4-44ec-8a65-6f1b69e57a9d",
                    ],
                    "X-Ms-Correlation-Request-Id": [
                        "45bdda71-4b5b-4b9b-9fd7-4df364bf7808",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260402T085811Z:45bdda71-4b5b-4b9b-9fd7-4df364bf7808",
                    ],
                    "Pragma": ["no-cache"],
                    "Content-Length": ["257"],
                    "Expires": ["-1"],
                    "X-Ms-Error-Code": ["ServiceUnavailable"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "X-Msedge-Ref": [
                        "Ref A: C36632182EBE4B8EB52F9106BF8C5BB2 Ref B: CHI261140502029 Ref C: 2026-04-02T08:58:02Z",
                    ],
                    "Retry-After": ["59"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "Date": [
                        "Thu, 02 Apr 2026 08:58:11 GMT",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0004e6480>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 83, 101, 114, 118, 105, 99, 101, 85, 110, 97, 118, 97, 105, 108, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 108, 117, 115, 116, 101, 114, 32, 105, 110, 115, 116, 97, 108, 108, 97, 116, 105, 111, 110, 32, 104, 97, 115, 32, 102, 97, 105, 108, 101, 100, 32, 100, 117, 101, 32, 116, 111, 32, 117, 110, 97, 118, 97, 105, 108, 97, 98, 108, 101, 32, 99, 97, 112, 97, 99, 105, 116, 121, 46, 32, 87, 104, 105, 108, 101, 32, 119, 101, 32, 97, 114, 101, 32, 98, 114, 105, 110, 103, 105, 110, 103, 32, 111, 110, 32, 97, 100, 100, 105, 116, 105, 111, ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:123]: Unexpected error:
    <*fmt.wrapError | 0xc0004e6700>: 
    failed starting cluster creation "basic-hcp-cluster2" in resourcegroup="customer-rg-p6dr85": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-p6dr85/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster2
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: ServiceUnavailable
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ServiceUnavailable",
        "message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed starting cluster creation \"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-p6dr85\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-p6dr85/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster2\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"ServiceUnavailable\",\n    \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc001402630>{
            ErrorCode: "ServiceUnavailable",
            StatusCode: 503,
            RawResponse: {
                Status: "503 Service Unavailable",
                StatusCode: 503,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Cache-Control": ["no-cache"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Request-Id": [
                        "981e3b54-276e-45f0-a48f-b05f015c79dc",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["8461"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/d750dcff-9cd4-44ec-8a65-6f1b69e57a9d",
                    ],
                    "X-Ms-Correlation-Request-Id": [
                        "45bdda71-4b5b-4b9b-9fd7-4df364bf7808",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260402T085811Z:45bdda71-4b5b-4b9b-9fd7-4df364bf7808",
                    ],
                    "Pragma": ["no-cache"],
                    "Content-Length": ["257"],
                    "Expires": ["-1"],
                    "X-Ms-Error-Code": ["ServiceUnavailable"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "X-Msedge-Ref": [
                        "Ref A: C36632182EBE4B8EB52F9106BF8C5BB2 Ref B: CHI261140502029 Ref C: 2026-04-02T08:58:02Z",
                    ],
                    "Retry-After": ["59"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "Date": [
                        "Thu, 02 Apr 2026 08:58:11 GMT",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0004e6480>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 83, 101, 114, 118, 105, 99, 101, 85, 110, 97, 118, 97, 105, 108, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 108, 117, 115, 116, 101, 114, 32, 105, 110, 115, 116, 97, 108, 108, 97, 116, 105, 111, 110, 32, 104, 97, 115, 32, 102, 97, 105, 108, 101, 100, 32, 100, 117, 101, 32, 116, 111, 32, 117, 110, 97, 118, 97, 105, 108, 97, 98, 108, 101, 32, 99, 97, 112, 97, 99, 105, 116, 121, 46, 32, 87, 104, 105, 108, 101, 32, 119, 101, 32, 97, 114, 101, 32, 98, 114, 105, 110, 103, 105, 110, 103, 32, 111, 110, 32, 97, 100, 100, 105, 116, 105, 111, ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error:
    <*fmt.wrapError | 0xc000880260>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-6tff8z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-6tff8z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: ServiceUnavailable
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ServiceUnavailable",
        "message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster basic-cluster: failed starting cluster creation \"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-6tff8z\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-6tff8z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"ServiceUnavailable\",\n    \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000880200>{
            msg: "failed starting cluster creation \"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-6tff8z\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-6tff8z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"ServiceUnavailable\",\n    \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0012ca1b0>{
                ErrorCode: "ServiceUnavailable",
                StatusCode: 503,
                RawResponse: {
                    Status: "503 Service Unavailable",
                    StatusCode: 503,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "2a87bdad-8340-4cda-a740-2d7cc278c733",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "294bd1e5-7abf-4f05-a3c2-931fa8a72c4e",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260402T085750Z:294bd1e5-7abf-4f05-a3c2-931fa8a72c4e",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Date": [
                            "Thu, 02 Apr 2026 08:57:50 GMT",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Envoy-Upstream-Service-Time": ["2649"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/cc9e8570-fcc2-4ba9-9d49-08bb67825537",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "X-Content-Type-Options": ["nosniff"],
                        "Content-Length": ["257"],
                        "Retry-After": ["59"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Ms-Fail...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error:
    <*fmt.wrapError | 0xc000880260>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-6tff8z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-6tff8z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: ServiceUnavailable
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ServiceUnavailable",
        "message": "Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster basic-cluster: failed starting cluster creation \"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-6tff8z\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-6tff8z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"ServiceUnavailable\",\n    \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000880200>{
            msg: "failed starting cluster creation \"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-6tff8z\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-6tff8z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: ServiceUnavailable\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"ServiceUnavailable\",\n    \"message\": \"Cluster installation has failed due to unavailable capacity. While we are bringing on additional capacity, try your cluster installation in another region or try again later\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0012ca1b0>{
                ErrorCode: "ServiceUnavailable",
                StatusCode: 503,
                RawResponse: {
                    Status: "503 Service Unavailable",
                    StatusCode: 503,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "2a87bdad-8340-4cda-a740-2d7cc278c733",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "294bd1e5-7abf-4f05-a3c2-931fa8a72c4e",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260402T085750Z:294bd1e5-7abf-4f05-a3c2-931fa8a72c4e",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Date": [
                            "Thu, 02 Apr 2026 08:57:50 GMT",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Envoy-Upstream-Service-Time": ["2649"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/cc9e8570-fcc2-4ba9-9d49-08bb67825537",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "X-Content-Type-Options": ["nosniff"],
                        "Content-Length": ["257"],
                        "Retry-After": ["59"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Ms-Fail...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-02 08:48	n/a	2039626069157351424

UpdateHCPCluster20251223 timed out after <minutes> minutes; context deadline exceeded

full failure pattern:

UpdateHCPCluster20251223 timed out after <minutes> minutes; context deadline exceeded

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:231]: Unexpected error:
    <*fmt.wrapErrors | 0xc0008ee840>: 
    failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-j9bcfb" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20251223 for cluster idms-e2e-hcp-cluster in resource group idms-j9bcfb, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-j9bcfb\" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20251223 for cluster idms-e2e-hcp-cluster in resource group idms-j9bcfb, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0008ba000>{
                s: "timeout '10.000000' minutes exceeded during UpdateHCPCluster20251223 for cluster idms-e2e-hcp-cluster in resource group idms-j9bcfb",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:231]: Unexpected error:
    <*fmt.wrapErrors | 0xc0008ee840>: 
    failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-j9bcfb" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20251223 for cluster idms-e2e-hcp-cluster in resource group idms-j9bcfb, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-j9bcfb\" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20251223 for cluster idms-e2e-hcp-cluster in resource group idms-j9bcfb, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0008ba000>{
                s: "timeout '10.000000' minutes exceeded during UpdateHCPCluster20251223 for cluster idms-e2e-hcp-cluster in resource group idms-j9bcfb",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-04 02:00	n/a	2040248180829851648

found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-<id>

full failure pattern:

found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-<id>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246]: Unexpected error:
    <*errors.joinError | 0xc00161a870>: 
    found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-8mw26m
    {
        errs: [
            <*errors.errorString | 0xc001950750>{
                s: "found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-8mw26m",
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246]: Unexpected error:
    <*errors.joinError | 0xc00161a870>: 
    found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-8mw26m
    {
        errs: [
            <*errors.errorString | 0xc001950750>{
                s: "found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-8mw26m",
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-01 22:51	n/a	2039475678092464128

Environment: PROD

E2E Jobs

Success Rate

32.35%

E2E success

32.35% (11/34)

Daily Run Outcomes

Successful runs Provision failures E2E failures Other failures

9 runs

1 run

1 run

13 runs

5 runs

1 run

4 runs

Pass rate	Runs	Date (UTC)	Suite	Test
62.86%	35	2026-04-07	prod/parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19
62.96%	27	2026-04-07	prod/parallel	Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest
64.86%	37	2026-04-07	prod/parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP
67.57%	37	2026-04-07	prod/parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle
70.27%	37	2026-04-07	prod/parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request

Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.

Jump to the full Failure Patterns view for this window

Failure pattern

Failed at

Trend

Also in

Interrupted by User

full failure pattern:

Interrupted by User

Signal:

e2e

DEV

Full failure examples (3)

fail [:0]: Interrupted by User
fail [:0]: Interrupted by User

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:184]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:184]: Interrupted by User

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:42]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:42]: Interrupted by User

Contributing tests (16)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling	4
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	4
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.22	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.23	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (18)

Date (UTC)	Associated PR	Prow job
2026-04-05 18:04	n/a	2040852983817703424
2026-04-05 15:03	n/a	2040807616363368448
2026-04-05 02:00	n/a	2040610572545822720
2026-04-03 18:45	n/a	2040138651320455168
2026-04-03 15:33	n/a	2040090253817024512
2026-04-03 06:22	n/a	2039951538536845312
2026-04-03 06:20	n/a	2039951072490950656
2026-04-03 02:00	n/a	2039885780922404864
2026-04-02 22:49	n/a	2039837667201388544
2026-04-02 21:22	n/a	2039815819470311424
2026-04-02 20:26	n/a	2039801784163110912
2026-04-02 18:38	n/a	2039774495035625472
2026-04-02 18:07	n/a	2039766803189600256
2026-04-02 16:29	n/a	2039741904471461888
2026-04-02 16:00	n/a	2039734796569022464
2026-04-02 15:56	n/a	2039733603084013568
2026-04-02 10:28	n/a	2039651240933068800
2026-03-30 06:09	n/a	2038498730092531712

Deserializaion Error: no output from command

full failure pattern:

Deserializaion Error: no output from command

Signal:

e2e

DEV

Full failure examples (3)

Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775406980 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should not be able to perform various invalid operations on cluster resources [PreLaunchSetup:None, Positivity:Negative, Importance:Medium, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:41
    STEP: creating a resource group @ 04/05/26 16:40:23.88
  "ts"="2026-04-05 16:40:23.880519" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-negative-tests-25748m"
    STEP: creating cluster parameters @ 04/05/26 16:40:28.315
    STEP: creating customer resources @ 04/05/26 16:40:28.315
    STEP: creating the HCP cluster @ 04/05/26 16:41:24.604
  "ts"="2026-04-05 16:41:24.604958" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="negative-tests-cluster" "resourceGroup"="rg-negative-tests-25748m"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775406980 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should not be able to perform various invalid operations on cluster resources [PreLaunchSetup:None, Positivity:Negative, Importance:Medium, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:41
    STEP: creating a resource group @ 04/05/26 16:40:23.88
  "ts"="2026-04-05 16:40:23.880519" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-negative-tests-25748m"
    STEP: creating cluster parameters @ 04/05/26 16:40:28.315
    STEP: creating customer resources @ 04/05/26 16:40:28.315
    STEP: creating the HCP cluster @ 04/05/26 16:41:24.604
  "ts"="2026-04-05 16:41:24.604958" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="negative-tests-cluster" "resourceGroup"="rg-negative-tests-25748m"
Command Error: signal: killed
Deserializaion Error: no output from command

Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775404643 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should be able to update node pool labels and taints [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
    STEP: creating a resource group @ 04/05/26 16:40:32.119
  "ts"="2026-04-05 16:40:32.119996" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-labels-taints-cbk9kk"
    STEP: creating cluster parameters @ 04/05/26 16:40:36.148
    STEP: creating customer resources @ 04/05/26 16:40:36.148
    STEP: creating the HCP cluster @ 04/05/26 16:41:32.398
  "ts"="2026-04-05 16:41:32.399116" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cluster-np-labels-taints" "resourceGroup"="rg-np-labels-taints-cbk9kk"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775404643 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should be able to update node pool labels and taints [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
    STEP: creating a resource group @ 04/05/26 16:40:32.119
  "ts"="2026-04-05 16:40:32.119996" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-labels-taints-cbk9kk"
    STEP: creating cluster parameters @ 04/05/26 16:40:36.148
    STEP: creating customer resources @ 04/05/26 16:40:36.148
    STEP: creating the HCP cluster @ 04/05/26 16:41:32.398
  "ts"="2026-04-05 16:41:32.399116" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cluster-np-labels-taints" "resourceGroup"="rg-np-labels-taints-cbk9kk"
Command Error: signal: killed
Deserializaion Error: no output from command

Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775404705 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231
    STEP: creating resource group @ 04/05/26 16:40:32.126
  "ts"="2026-04-05 16:40:32.126074" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-5tv55j-92x7wf"
    STEP: creating cluster parameters at control plane version @ 04/05/26 16:40:35.94
    STEP: creating customer resources @ 04/05/26 16:40:36.059
    STEP: creating the HCP cluster with version 4.21.7 @ 04/05/26 16:41:32.412
  "ts"="2026-04-05 16:41:32.413023" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-5tv55j" "resourceGroup"="rg-np-version-upgrade-5tv55j-92x7wf"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775404705 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231
    STEP: creating resource group @ 04/05/26 16:40:32.126
  "ts"="2026-04-05 16:40:32.126074" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-5tv55j-92x7wf"
    STEP: creating cluster parameters at control plane version @ 04/05/26 16:40:35.94
    STEP: creating customer resources @ 04/05/26 16:40:36.059
    STEP: creating the HCP cluster with version 4.21.7 @ 04/05/26 16:41:32.412
  "ts"="2026-04-05 16:41:32.413023" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-5tv55j" "resourceGroup"="rg-np-version-upgrade-5tv55j-92x7wf"
Command Error: signal: killed
Deserializaion Error: no output from command

Contributing tests (16)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest	6
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to update node pool labels and taints	5
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1

Affected runs (9)

Date (UTC)	Associated PR	Prow job
2026-04-05 15:03	n/a	2040807616363368448
2026-04-03 18:45	n/a	2040138651320455168
2026-04-03 15:33	n/a	2040090253817024512
2026-04-03 06:20	n/a	2039951072490950656
2026-04-02 22:49	n/a	2039837667201388544
2026-04-02 21:22	n/a	2039815819470311424
2026-04-02 19:11	n/a	2039782869789839360
2026-04-02 18:07	n/a	2039766803189600256
2026-04-02 00:18	n/a	2039497738340536320

timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds

full failure pattern:

timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds

Signal:

e2e

INT, STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:126]: Unexpected error:
    <*fmt.wrapErrors | 0xc000eb6a20>: 
    failed waiting for hcpCluster="cilium-cluster" in resourcegroup="e2e-cilium-l56s4b" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-l56s4b, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cilium-cluster\" in resourcegroup=\"e2e-cilium-l56s4b\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-l56s4b, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000905dd0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-l56s4b",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:126]: Unexpected error:
    <*fmt.wrapErrors | 0xc000eb6a20>: 
    failed waiting for hcpCluster="cilium-cluster" in resourcegroup="e2e-cilium-l56s4b" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-l56s4b, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cilium-cluster\" in resourcegroup=\"e2e-cilium-l56s4b\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-l56s4b, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000905dd0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-l56s4b",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011b0810>: 
    failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-2v4phv" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-2v4phv, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-2v4phv\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-2v4phv, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009a5550>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-2v4phv",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011b0810>: 
    failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-2v4phv" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-2v4phv, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-2v4phv\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-2v4phv, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009a5550>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-2v4phv",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc0013f0840>: 
    failed waiting for hcpCluster="cluster-zstream-4-19-f4hz5d" in resourcegroup="rg-zstream-upgrade-4-19-mf7hcn" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-f4hz5d in resource group rg-zstream-upgrade-4-19-mf7hcn, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-f4hz5d\" in resourcegroup=\"rg-zstream-upgrade-4-19-mf7hcn\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-f4hz5d in resource group rg-zstream-upgrade-4-19-mf7hcn, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0015a34a0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-f4hz5d in resource group rg-zstream-upgrade-4-19-mf7hcn",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc0013f0840>: 
    failed waiting for hcpCluster="cluster-zstream-4-19-f4hz5d" in resourcegroup="rg-zstream-upgrade-4-19-mf7hcn" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-f4hz5d in resource group rg-zstream-upgrade-4-19-mf7hcn, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-f4hz5d\" in resourcegroup=\"rg-zstream-upgrade-4-19-mf7hcn\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-f4hz5d in resource group rg-zstream-upgrade-4-19-mf7hcn, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0015a34a0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-f4hz5d in resource group rg-zstream-upgrade-4-19-mf7hcn",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

Contributing tests (14)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19	6
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	5
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.19	5
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	5
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	4
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.20	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1

Affected runs (8)

Date (UTC)	Associated PR	Prow job
2026-04-03 18:45	n/a	2040138651320455168
2026-04-03 15:33	n/a	2040090253817024512
2026-04-03 06:22	n/a	2039951538536845312
2026-04-03 06:20	n/a	2039951072490950656
2026-04-02 22:49	n/a	2039837667201388544
2026-04-02 21:22	n/a	2039815819470311424
2026-04-02 20:26	n/a	2039801784163110912
2026-04-02 19:11	n/a	2039782869789839360

ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift

Signal:

e2e

STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error:
    <*fmt.wrapError | 0xc000dfab80>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-xhmz6z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-xhmz6z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0014f4090>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Cache-Control": ["no-cache"],
                    "Pragma": ["no-cache"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Request-Id": [
                        "3a55fa98-acad-4ad6-a240-181bf4b8ec64",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Correlation-Request-Id": [
                        "ca924b49-6e46-45a9-868d-228ae0481199",
                    ],
                    "X-Msedge-Ref": [
                        "Ref A: 07C6F1DD1B8E413285DC76D57E2C9826 Ref B: CHI261140507060 Ref C: 2026-04-05T20:51:26Z",
                    ],
                    "Date": [
                        "Sun, 05 Apr 2026 20:51:26 GMT",
                    ],
                    "Content-Length": ["107"],
                    "X-Envoy-Upstream-Service-Time": ["302"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260405T205126Z:ca924b49-6e46-45a9-868d-228ae0481199",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "Expires": ["-1"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/36803a68-b504-4c65-a064-df40a0d32fe8",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000dfaaa0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "management.azu...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error:
    <*fmt.wrapError | 0xc000dfab80>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-xhmz6z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-xhmz6z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0014f4090>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Cache-Control": ["no-cache"],
                    "Pragma": ["no-cache"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Request-Id": [
                        "3a55fa98-acad-4ad6-a240-181bf4b8ec64",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Correlation-Request-Id": [
                        "ca924b49-6e46-45a9-868d-228ae0481199",
                    ],
                    "X-Msedge-Ref": [
                        "Ref A: 07C6F1DD1B8E413285DC76D57E2C9826 Ref B: CHI261140507060 Ref C: 2026-04-05T20:51:26Z",
                    ],
                    "Date": [
                        "Sun, 05 Apr 2026 20:51:26 GMT",
                    ],
                    "Content-Length": ["107"],
                    "X-Envoy-Upstream-Service-Time": ["302"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260405T205126Z:ca924b49-6e46-45a9-868d-228ae0481199",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "Expires": ["-1"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/36803a68-b504-4c65-a064-df40a0d32fe8",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000dfaaa0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "management.azu...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:155]: Unexpected error:
    <*fmt.wrapError | 0xc000aa6120>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-j9zwrb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-j9zwrb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc00093e060>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Cache-Control": ["no-cache"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/6761ba3e-e3d5-4f2b-b121-1aaf18258695",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "Pragma": ["no-cache"],
                    "X-Envoy-Upstream-Service-Time": ["158"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260405T205956Z:d0b57af8-bf86-42e3-81f6-8c47f1266249",
                    ],
                    "Date": [
                        "Sun, 05 Apr 2026 20:59:56 GMT",
                    ],
                    "Content-Length": ["107"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Msedge-Ref": [
                        "Ref A: 895D9C70FEA44594A28EC00C03902856 Ref B: CHI261140509062 Ref C: 2026-04-05T20:59:56Z",
                    ],
                    "Expires": ["-1"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Request-Id": [
                        "126ca117-9440-42c5-af31-856f795dd47d",
                    ],
                    "X-Ms-Correlation-Request-Id": [
                        "d0b57af8-bf86-42e3-81f6-8c47f1266249",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000aa6060>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "m...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:155]: Unexpected error:
    <*fmt.wrapError | 0xc000aa6120>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-j9zwrb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-j9zwrb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc00093e060>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Cache-Control": ["no-cache"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/6761ba3e-e3d5-4f2b-b121-1aaf18258695",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "Pragma": ["no-cache"],
                    "X-Envoy-Upstream-Service-Time": ["158"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260405T205956Z:d0b57af8-bf86-42e3-81f6-8c47f1266249",
                    ],
                    "Date": [
                        "Sun, 05 Apr 2026 20:59:56 GMT",
                    ],
                    "Content-Length": ["107"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Msedge-Ref": [
                        "Ref A: 895D9C70FEA44594A28EC00C03902856 Ref B: CHI261140509062 Ref C: 2026-04-05T20:59:56Z",
                    ],
                    "Expires": ["-1"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Request-Id": [
                        "126ca117-9440-42c5-af31-856f795dd47d",
                    ],
                    "X-Ms-Correlation-Request-Id": [
                        "d0b57af8-bf86-42e3-81f6-8c47f1266249",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000aa6060>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "m...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error:
    <*fmt.wrapError | 0xc00064c5a0>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-d2x786/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-d2x786/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc001695c50>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/37faf447-4cbb-45d8-ab1c-8f7f75feed59",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Content-Type-Options": ["nosniff"],
                    "Cache-Control": ["no-cache"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["1651"],
                    "Pragma": ["no-cache"],
                    "Content-Length": ["107"],
                    "Expires": ["-1"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260405T210827Z:1be05225-58f9-4e68-ae82-4585f065edaf",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "X-Ms-Request-Id": [
                        "f03a6005-7553-4eb6-9d49-9efe378f7a0f",
                    ],
                    "X-Ms-Correlation-Request-Id": [
                        "1be05225-58f9-4e68-ae82-4585f065edaf",
                    ],
                    "X-Msedge-Ref": [
                        "Ref A: E4DC784654E84C74A2C16AB09A9371E1 Ref B: CHI261140508023 Ref C: 2026-04-05T21:08:25Z",
                    ],
                    "Date": [
                        "Sun, 05 Apr 2026 21:08:26 GMT",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc00064c380>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
          ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error:
    <*fmt.wrapError | 0xc00064c5a0>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-d2x786/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-d2x786/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc001695c50>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/37faf447-4cbb-45d8-ab1c-8f7f75feed59",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Content-Type-Options": ["nosniff"],
                    "Cache-Control": ["no-cache"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["1651"],
                    "Pragma": ["no-cache"],
                    "Content-Length": ["107"],
                    "Expires": ["-1"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260405T210827Z:1be05225-58f9-4e68-ae82-4585f065edaf",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "X-Ms-Request-Id": [
                        "f03a6005-7553-4eb6-9d49-9efe378f7a0f",
                    ],
                    "X-Ms-Correlation-Request-Id": [
                        "1be05225-58f9-4e68-ae82-4585f065edaf",
                    ],
                    "X-Msedge-Ref": [
                        "Ref A: E4DC784654E84C74A2C16AB09A9371E1 Ref B: CHI261140508023 Ref C: 2026-04-05T21:08:25Z",
                    ],
                    "Date": [
                        "Sun, 05 Apr 2026 21:08:26 GMT",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc00064c380>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
          ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Contributing tests (12)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	4
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1

Affected runs (7)

Date (UTC)	Associated PR	Prow job
2026-04-05 20:06	n/a	2040883810354073600
2026-04-03 18:45	n/a	2040138651320455168
2026-04-03 15:33	n/a	2040090253817024512
2026-04-03 06:22	n/a	2039951538536845312
2026-04-02 22:49	n/a	2039837667201388544
2026-04-02 21:22	n/a	2039815819470311424
2026-04-02 20:26	n/a	2039801784163110912

timeout during CreateHCPClusterAndWait; context deadline exceeded

full failure pattern:

timeout during CreateHCPClusterAndWait; context deadline exceeded

Signal:

e2e

DEV, INT, STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a712f0>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-x5gxtd, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-x5gxtd" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-x5gxtd, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-x5gxtd, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-x5gxtd\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-x5gxtd, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000f71180>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-x5gxtd",
            },
            <*fmt.wrapErrors | 0xc000a712c0>{
                msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-x5gxtd\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-x5gxtd, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000f71180>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-x5gxtd",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a712f0>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-x5gxtd, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-x5gxtd" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-x5gxtd, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-x5gxtd, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-x5gxtd\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-x5gxtd, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000f71180>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-x5gxtd",
            },
            <*fmt.wrapErrors | 0xc000a712c0>{
                msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-x5gxtd\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-x5gxtd, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000f71180>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-x5gxtd",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ff2060>: 
    failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-bg5ssk, error: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-bg5ssk" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-bg5ssk, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-bg5ssk, error: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-bg5ssk\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-bg5ssk, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0016a4970>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-bg5ssk",
            },
            <*fmt.wrapErrors | 0xc000ff2030>{
                msg: "failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-bg5ssk\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-bg5ssk, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0016a4970>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-bg5ssk",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ff2060>: 
    failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-bg5ssk, error: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-bg5ssk" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-bg5ssk, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-bg5ssk, error: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-bg5ssk\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-bg5ssk, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0016a4970>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-bg5ssk",
            },
            <*fmt.wrapErrors | 0xc000ff2030>{
                msg: "failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-bg5ssk\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-bg5ssk, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0016a4970>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-bg5ssk",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f36a20>: 
    failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-29zthj, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-29zthj" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-29zthj, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-29zthj, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-29zthj\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-29zthj, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0013c5050>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-29zthj",
            },
            <*fmt.wrapErrors | 0xc000f369f0>{
                msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-29zthj\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-29zthj, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0013c5050>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-29zthj",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f36a20>: 
    failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-29zthj, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-29zthj" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-29zthj, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-29zthj, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-29zthj\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-29zthj, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0013c5050>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-29zthj",
            },
            <*fmt.wrapErrors | 0xc000f369f0>{
                msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-29zthj\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-29zthj, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0013c5050>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-29zthj",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

Contributing tests (25)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.19	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should create an HCP cluster and validate TLS certificates	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	1

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-04-05 15:03	n/a	2040807616363368448
2026-04-02 00:18	n/a	2039497738340536320
2026-03-31 02:00	n/a	2038798596630908928
2026-03-30 06:09	n/a	2038498730092531712
2026-03-30 00:47	n/a	2038417864989347840

Process did not finish before 2h0m0s timeout

full failure pattern:

Process did not finish before 2h0m0s timeout

Signal:

unknown

none

Full failure examples (3)

{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-05T17:06:11Z"}

{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-03T20:48:28Z"}

{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-03T08:22:36Z"}

Contributing tests (1)

Failed At	Job	Test	Support
unknown	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Job run should complete before timeout	5

Affected runs (5)

Date (UTC)	Associated PR	Prow job
2026-04-05 15:03	n/a	2040807616363368448
2026-04-03 18:45	n/a	2040138651320455168
2026-04-03 06:20	n/a	2039951072490950656
2026-04-02 22:49	n/a	2039837667201388544
2026-04-02 00:18	n/a	2039497738340536320

Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed

full failure pattern:

Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:153]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:153]: Cluster provisioning failed

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	2

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-04-02 00:18	n/a	2039497738340536320
2026-03-30 06:09	n/a	2038498730092531712

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

Signal:

e2e

STG

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapError | 0xc000e0b7c0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-l2s4f2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02",
      "name": "f9df063e-f124-4f77-8a25-358c9d5cde02",
      "status": "Failed",
      "startTime": "2026-03-30T06:16:32.010035382Z",
      "endTime": "2026-03-30T06:16:48.681254232Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-l2s4f2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\",\n  \"name\": \"f9df063e-f124-4f77-8a25-358c9d5cde02\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:16:32.010035382Z\",\n  \"endTime\": \"2026-03-30T06:16:48.681254232Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000e0b7a0>{
            msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-l2s4f2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\",\n  \"name\": \"f9df063e-f124-4f77-8a25-358c9d5cde02\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:16:32.010035382Z\",\n  \"endTime\": \"2026-03-30T06:16:48.681254232Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00104d650>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T061652Z:5fa8b3d2-fffd-493a-9339-7cb51375d257",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "Date": [
                            "Mon, 30 Mar 2026 06:16:51 GMT",
                        ],
                        "Expires": ["-1"],
                        "X-Msedge-Ref": [
                            "Ref A: 9C0B547CA14B41D4A16FDC03FE815989 Ref B: CHI261140514036 Ref C: 2026-03-30T06:16:52Z",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
      ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapError | 0xc000e0b7c0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-l2s4f2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02",
      "name": "f9df063e-f124-4f77-8a25-358c9d5cde02",
      "status": "Failed",
      "startTime": "2026-03-30T06:16:32.010035382Z",
      "endTime": "2026-03-30T06:16:48.681254232Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-l2s4f2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\",\n  \"name\": \"f9df063e-f124-4f77-8a25-358c9d5cde02\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:16:32.010035382Z\",\n  \"endTime\": \"2026-03-30T06:16:48.681254232Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000e0b7a0>{
            msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-l2s4f2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\",\n  \"name\": \"f9df063e-f124-4f77-8a25-358c9d5cde02\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:16:32.010035382Z\",\n  \"endTime\": \"2026-03-30T06:16:48.681254232Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00104d650>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T061652Z:5fa8b3d2-fffd-493a-9339-7cb51375d257",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "Date": [
                            "Mon, 30 Mar 2026 06:16:51 GMT",
                        ],
                        "Expires": ["-1"],
                        "X-Msedge-Ref": [
                            "Ref A: 9C0B547CA14B41D4A16FDC03FE815989 Ref B: CHI261140514036 Ref C: 2026-03-30T06:16:52Z",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
      ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: Unexpected error:
    <*fmt.wrapError | 0xc000a0b040>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-t4849k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395",
      "name": "671489be-ab4c-40e2-81b0-625f59cf6395",
      "status": "Failed",
      "startTime": "2026-03-30T06:22:24.890658519Z",
      "endTime": "2026-03-30T06:22:55.769894295Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-t4849k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\",\n  \"name\": \"671489be-ab4c-40e2-81b0-625f59cf6395\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:22:24.890658519Z\",\n  \"endTime\": \"2026-03-30T06:22:55.769894295Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a0b020>{
            msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-t4849k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\",\n  \"name\": \"671489be-ab4c-40e2-81b0-625f59cf6395\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:22:24.890658519Z\",\n  \"endTime\": \"2026-03-30T06:22:55.769894295Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0011f0ed0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/be562cea-8623-4678-adaa-0057ff53bbd9",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "8b2df379-7ba3-4320-8e92-21a566e40b6c",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: BC420E598D454882AFE3F4518AA17A00 Ref B: CHI261140512052 Ref C: 2026-03-30T06:22:55Z",
                        ],
                        "Content-Length": ["503"],
                        "X-Ms-Request-Id": [
                            "fef2e839-309a-4189-8dfd-f18f57fe8ccf",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Rema...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: Unexpected error:
    <*fmt.wrapError | 0xc000a0b040>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-t4849k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395",
      "name": "671489be-ab4c-40e2-81b0-625f59cf6395",
      "status": "Failed",
      "startTime": "2026-03-30T06:22:24.890658519Z",
      "endTime": "2026-03-30T06:22:55.769894295Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-t4849k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\",\n  \"name\": \"671489be-ab4c-40e2-81b0-625f59cf6395\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:22:24.890658519Z\",\n  \"endTime\": \"2026-03-30T06:22:55.769894295Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a0b020>{
            msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-t4849k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\",\n  \"name\": \"671489be-ab4c-40e2-81b0-625f59cf6395\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:22:24.890658519Z\",\n  \"endTime\": \"2026-03-30T06:22:55.769894295Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0011f0ed0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/be562cea-8623-4678-adaa-0057ff53bbd9",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "8b2df379-7ba3-4320-8e92-21a566e40b6c",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: BC420E598D454882AFE3F4518AA17A00 Ref B: CHI261140512052 Ref C: 2026-03-30T06:22:55Z",
                        ],
                        "Content-Length": ["503"],
                        "X-Ms-Request-Id": [
                            "fef2e839-309a-4189-8dfd-f18f57fe8ccf",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Rema...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: Unexpected error:
    <*fmt.wrapError | 0xc0016131a0>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-8qw9d2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e",
      "name": "e8c3c061-19f1-4d72-b474-3fbe98891e9e",
      "status": "Failed",
      "startTime": "2026-03-30T06:18:33.483509413Z",
      "endTime": "2026-03-30T06:18:44.018298845Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8qw9d2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n  \"name\": \"e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:18:33.483509413Z\",\n  \"endTime\": \"2026-03-30T06:18:44.018298845Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc001613180>{
            msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8qw9d2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n  \"name\": \"e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:18:33.483509413Z\",\n  \"endTime\": \"2026-03-30T06:18:44.018298845Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00157d260>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Routing-Request-Id": [
                            "CANADAEAST:20260330T061854Z:72e130e2-01fe-4e82-86ac-3810c7b0b18a",
                        ],
                        "Date": [
                            "Mon, 30 Mar 2026 06:18:53 GMT",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "72e130e2-01fe-4e82-86ac-3810c7b0b18a",
                        ],
                        "Pragma": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: 05A3F815056F4198B1B5D4420C306AED Ref B: CHI261140504023 Ref C: 2026-03-30T06:18:53Z",
                        ],
                        "Cache-Control": ["no-cache"],
                        "X-Ms-Request-Id": [
                            "111c62...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: Unexpected error:
    <*fmt.wrapError | 0xc0016131a0>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-8qw9d2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e",
      "name": "e8c3c061-19f1-4d72-b474-3fbe98891e9e",
      "status": "Failed",
      "startTime": "2026-03-30T06:18:33.483509413Z",
      "endTime": "2026-03-30T06:18:44.018298845Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8qw9d2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n  \"name\": \"e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:18:33.483509413Z\",\n  \"endTime\": \"2026-03-30T06:18:44.018298845Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc001613180>{
            msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8qw9d2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n  \"name\": \"e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:18:33.483509413Z\",\n  \"endTime\": \"2026-03-30T06:18:44.018298845Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00157d260>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Routing-Request-Id": [
                            "CANADAEAST:20260330T061854Z:72e130e2-01fe-4e82-86ac-3810c7b0b18a",
                        ],
                        "Date": [
                            "Mon, 30 Mar 2026 06:18:53 GMT",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "72e130e2-01fe-4e82-86ac-3810c7b0b18a",
                        ],
                        "Pragma": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: 05A3F815056F4198B1B5D4420C306AED Ref B: CHI261140504023 Ref C: 2026-03-30T06:18:53Z",
                        ],
                        "Cache-Control": ["no-cache"],
                        "X-Ms-Request-Id": [
                            "111c62...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Contributing tests (14)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should create an HCP cluster and validate TLS certificates	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-03-30 06:09	n/a	2038498730092531712

ERROR CODE: InternalServerError; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microso...

full failure pattern:

ERROR CODE: InternalServerError; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapError | 0xc0010502c0>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-8xlw7k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3",
      "name": "3484e971-8275-4ae3-a76f-65d592316cf3",
      "status": "Failed",
      "startTime": "2026-04-02T01:29:14.718319727Z",
      "endTime": "2026-04-02T01:30:18.264779861Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b531933373' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-8xlw7k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3\",\n  \"name\": \"3484e971-8275-4ae3-a76f-65d592316cf3\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T01:29:14.718319727Z\",\n  \"endTime\": \"2026-04-02T01:30:18.264779861Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b531933373' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0010502a0>{
            msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-8xlw7k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3\",\n  \"name\": \"3484e971-8275-4ae3-a76f-65d592316cf3\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T01:29:14.718319727Z\",\n  \"endTime\": \"2026-04-02T01:30:18.264779861Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b5319333...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapError | 0xc0010502c0>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-8xlw7k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3",
      "name": "3484e971-8275-4ae3-a76f-65d592316cf3",
      "status": "Failed",
      "startTime": "2026-04-02T01:29:14.718319727Z",
      "endTime": "2026-04-02T01:30:18.264779861Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b531933373' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-8xlw7k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3\",\n  \"name\": \"3484e971-8275-4ae3-a76f-65d592316cf3\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T01:29:14.718319727Z\",\n  \"endTime\": \"2026-04-02T01:30:18.264779861Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b531933373' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0010502a0>{
            msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-8xlw7k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3484e971-8275-4ae3-a76f-65d592316cf3\",\n  \"name\": \"3484e971-8275-4ae3-a76f-65d592316cf3\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T01:29:14.718319727Z\",\n  \"endTime\": \"2026-04-02T01:30:18.264779861Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8xlw7k/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b5319333...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapError | 0xc00078d420>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-p4zvhp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29",
      "name": "f32ace59-a2fb-42f6-9421-b40f5132fd29",
      "status": "Failed",
      "startTime": "2026-04-02T00:27:03.073725945Z",
      "endTime": "2026-04-02T00:27:41.510479828Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-p4zvhp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29\",\n  \"name\": \"f32ace59-a2fb-42f6-9421-b40f5132fd29\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T00:27:03.073725945Z\",\n  \"endTime\": \"2026-04-02T00:27:41.510479828Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00078d3c0>{
            msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-p4zvhp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29\",\n  \"name\": \"f32ace59-a2fb-42f6-9421-b40f5132fd29\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T00:27:03.073725945Z\",\n  \"endTime\": \"2026-04-02T00:27:41.510479828Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not ha...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapError | 0xc00078d420>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-p4zvhp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29",
      "name": "f32ace59-a2fb-42f6-9421-b40f5132fd29",
      "status": "Failed",
      "startTime": "2026-04-02T00:27:03.073725945Z",
      "endTime": "2026-04-02T00:27:41.510479828Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-p4zvhp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29\",\n  \"name\": \"f32ace59-a2fb-42f6-9421-b40f5132fd29\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T00:27:03.073725945Z\",\n  \"endTime\": \"2026-04-02T00:27:41.510479828Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00078d3c0>{
            msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-p4zvhp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32ace59-a2fb-42f6-9421-b40f5132fd29\",\n  \"name\": \"f32ace59-a2fb-42f6-9421-b40f5132fd29\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T00:27:03.073725945Z\",\n  \"endTime\": \"2026-04-02T00:27:41.510479828Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-p4zvhp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not ha...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
    <*fmt.wrapError | 0xc000894f20>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-8sxmq6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587",
      "name": "a300f2ba-df79-4404-a4a1-aae22a4d2587",
      "status": "Failed",
      "startTime": "2026-04-02T00:22:54.935266426Z",
      "endTime": "2026-04-02T00:23:25.288802016Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-8sxmq6\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587\",\n  \"name\": \"a300f2ba-df79-4404-a4a1-aae22a4d2587\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T00:22:54.935266426Z\",\n  \"endTime\": \"2026-04-02T00:23:25.288802016Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000894f00>{
            msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-8sxmq6\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587\",\n  \"name\": \"a300f2ba-df79-4404-a4a1-aae22a4d2587\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T00:22:54.935266426Z\",\n  \"endTime\": \"2026-04-02T00:23:25.288802016Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authori...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
    <*fmt.wrapError | 0xc000894f20>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-8sxmq6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587",
      "name": "a300f2ba-df79-4404-a4a1-aae22a4d2587",
      "status": "Failed",
      "startTime": "2026-04-02T00:22:54.935266426Z",
      "endTime": "2026-04-02T00:23:25.288802016Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-8sxmq6\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587\",\n  \"name\": \"a300f2ba-df79-4404-a4a1-aae22a4d2587\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T00:22:54.935266426Z\",\n  \"endTime\": \"2026-04-02T00:23:25.288802016Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000894f00>{
            msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-8sxmq6\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a300f2ba-df79-4404-a4a1-aae22a4d2587\",\n  \"name\": \"a300f2ba-df79-4404-a4a1-aae22a4d2587\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T00:22:54.935266426Z\",\n  \"endTime\": \"2026-04-02T00:23:25.288802016Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8sxmq6/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client '05d38701-f93a-4033-9e9e-2d387e154e87' with object id '3bb4466b-2c3a-411a-bdf6-ab9062e5a2e8' does not have authori...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-02 00:18	n/a	2039497738340536320

Cluster should become ready within 45 minutes

full failure pattern:

Cluster should become ready within 45 minutes

Signal:

e2e

DEV

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.004s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.004s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-05 15:03	n/a	2040807616363368448

found 1 managed resource groups left behind HCP clusters in cluster-listing-<id>

full failure pattern:

found 1 managed resource groups left behind HCP clusters in cluster-listing-<id>

Signal:

e2e

INT

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246]: Unexpected error:
    <*errors.joinError | 0xc001010078>: 
    found 1 managed resource groups left behind HCP clusters in cluster-listing-g7w797
    {
        errs: [
            <*errors.errorString | 0xc00185b800>{
                s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-g7w797",
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246]: Unexpected error:
    <*errors.joinError | 0xc001010078>: 
    found 1 managed resource groups left behind HCP clusters in cluster-listing-g7w797
    {
        errs: [
            <*errors.errorString | 0xc00185b800>{
                s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-g7w797",
            },
        ],
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to list HCP clusters without node pools at the resource group level	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-02 21:22	n/a	2039815819470311424

ERROR CODE: DeploymentNotFound

full failure pattern:

ERROR CODE: DeploymentNotFound

Signal:

e2e

DEV

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
    <*fmt.wrapError | 0xc0010281e0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-subnet-reuse-fnhchx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-subnet-reuse-fnhchx/operationStatuses/08584264532945813091
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: DeploymentNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentNotFound",
        "message": "Deployment 'mi-cluster-subnet-reuse-fnhchx' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-subnet-reuse-fnhchx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-subnet-reuse-fnhchx/operationStatuses/08584264532945813091\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-cluster-subnet-reuse-fnhchx' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0010281a0>{
            msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-subnet-reuse-fnhchx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-subnet-reuse-fnhchx/operationStatuses/08584264532945813091\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-cluster-subnet-reuse-fnhchx' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*fmt.wrapError | 0xc001028180>{
                msg: "failed waiting for deployment \"mi-cluster-subnet-reuse-fnhchx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-subnet-reuse-fnhchx/operationStatuses/08584264532945813091\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-cluster-subnet-reuse-fnhchx' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*exported.ResponseError | 0xc0006421b0>{
                    ErrorCode: "DeploymentNotFound",
                    StatusCode: 404,
                    RawResponse: {
                        Status: "404 Not Found",
                        StatusCode: 404,
                        Proto: "HTTP/2.0",
                        ProtoMajor: 2,
                        ProtoMinor: 0,
                        Header: {
                            "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"],
                            "Date": [
                                "Thu, 02 Apr 2026 18:13:11 GMT",
                            ],
                            "X-Ms-Failure-Cause": ["gateway"],
                            "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"],
                            "X-Ms-Request-Id": [
                                "4af1ec23-5e77-436c-a930-c88c039bbaf2",
                            ],
                            "X-Ms-Correlation-Request-Id": [
                                "4af1ec23-5e77-436c-a930-c88c039bbaf2",
                            ],
                            "Strict-Transport-Security": [
                                "max-age=31536000; includeSubDomains",
                            ],
                            "Cache-Control": ["no-cache"],
                            "Pragma": ["no-cache"],
                            "Content-Length": ["115"],
                 ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
    <*fmt.wrapError | 0xc0010281e0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-subnet-reuse-fnhchx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-subnet-reuse-fnhchx/operationStatuses/08584264532945813091
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: DeploymentNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentNotFound",
        "message": "Deployment 'mi-cluster-subnet-reuse-fnhchx' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-subnet-reuse-fnhchx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-subnet-reuse-fnhchx/operationStatuses/08584264532945813091\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-cluster-subnet-reuse-fnhchx' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0010281a0>{
            msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-subnet-reuse-fnhchx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-subnet-reuse-fnhchx/operationStatuses/08584264532945813091\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-cluster-subnet-reuse-fnhchx' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*fmt.wrapError | 0xc001028180>{
                msg: "failed waiting for deployment \"mi-cluster-subnet-reuse-fnhchx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-subnet-reuse-fnhchx/operationStatuses/08584264532945813091\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'mi-cluster-subnet-reuse-fnhchx' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*exported.ResponseError | 0xc0006421b0>{
                    ErrorCode: "DeploymentNotFound",
                    StatusCode: 404,
                    RawResponse: {
                        Status: "404 Not Found",
                        StatusCode: 404,
                        Proto: "HTTP/2.0",
                        ProtoMajor: 2,
                        ProtoMinor: 0,
                        Header: {
                            "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"],
                            "Date": [
                                "Thu, 02 Apr 2026 18:13:11 GMT",
                            ],
                            "X-Ms-Failure-Cause": ["gateway"],
                            "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"],
                            "X-Ms-Request-Id": [
                                "4af1ec23-5e77-436c-a930-c88c039bbaf2",
                            ],
                            "X-Ms-Correlation-Request-Id": [
                                "4af1ec23-5e77-436c-a930-c88c039bbaf2",
                            ],
                            "Strict-Transport-Security": [
                                "max-age=31536000; includeSubDomains",
                            ],
                            "Cache-Control": ["no-cache"],
                            "Pragma": ["no-cache"],
                            "Content-Length": ["115"],
                 ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-02 18:07	n/a	2039766803189600256

ERROR CODE: InternalServerError; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microso...

full failure pattern:

ERROR CODE: InternalServerError; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-<id>' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: Unexpected error:
    <*fmt.wrapError | 0xc000bea800>: 
    failed to create HCP cluster np-version-upgrade-cluster-z8jp8j: failed waiting for cluster="np-version-upgrade-cluster-z8jp8j" in resourcegroup="rg-np-version-upgrade-z8jp8j-brqz27" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0",
      "name": "3ad28075-f0f4-45dd-b143-f32346eb45d0",
      "status": "Failed",
      "startTime": "2026-04-02T01:33:37.647438148Z",
      "endTime": "2026-04-02T01:34:08.890577638Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg-np-upgrade-z8jp8j' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b531933373' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster np-version-upgrade-cluster-z8jp8j: failed waiting for cluster=\"np-version-upgrade-cluster-z8jp8j\" in resourcegroup=\"rg-np-version-upgrade-z8jp8j-brqz27\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0\",\n  \"name\": \"3ad28075-f0f4-45dd-b143-f32346eb45d0\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T01:33:37.647438148Z\",\n  \"endTime\": \"2026-04-02T01:34:08.890577638Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg-np-upgrade-z8jp8j' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b531933373' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000bea7c0>{
            msg: "failed waiting for cluster=\"np-version-upgrade-cluster-z8jp8j\" in resourcegroup=\"rg-np-version-upgrade-z8jp8j-brqz27\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0\",\n  \"name\": \"3ad28075-f0f4-45dd-b143-f32346eb45d0\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T01:33:37.647438148Z\",\n  \"endTime\": \"2026-04-02T01:34:08.890577638Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg-np-upgrade-z8jp8j' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:132]: Unexpected error:
    <*fmt.wrapError | 0xc000bea800>: 
    failed to create HCP cluster np-version-upgrade-cluster-z8jp8j: failed waiting for cluster="np-version-upgrade-cluster-z8jp8j" in resourcegroup="rg-np-version-upgrade-z8jp8j-brqz27" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0",
      "name": "3ad28075-f0f4-45dd-b143-f32346eb45d0",
      "status": "Failed",
      "startTime": "2026-04-02T01:33:37.647438148Z",
      "endTime": "2026-04-02T01:34:08.890577638Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg-np-upgrade-z8jp8j' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b531933373' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster np-version-upgrade-cluster-z8jp8j: failed waiting for cluster=\"np-version-upgrade-cluster-z8jp8j\" in resourcegroup=\"rg-np-version-upgrade-z8jp8j-brqz27\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0\",\n  \"name\": \"3ad28075-f0f4-45dd-b143-f32346eb45d0\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T01:33:37.647438148Z\",\n  \"endTime\": \"2026-04-02T01:34:08.890577638Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg-np-upgrade-z8jp8j' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client 'dc7ea24e-36a3-498b-8f49-89d5b180a0d0' with object id 'cc5bfbf9-edb5-415d-8cef-d5b531933373' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000bea7c0>{
            msg: "failed waiting for cluster=\"np-version-upgrade-cluster-z8jp8j\" in resourcegroup=\"rg-np-version-upgrade-z8jp8j-brqz27\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ad28075-f0f4-45dd-b143-f32346eb45d0\",\n  \"name\": \"3ad28075-f0f4-45dd-b143-f32346eb45d0\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T01:33:37.647438148Z\",\n  \"endTime\": \"2026-04-02T01:34:08.890577638Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg-np-upgrade-z8jp8j' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z8jp8j-brqz27/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-np-upgrade-z8jp8j\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-02 00:18	n/a	2039497738340536320

ERROR CODE: InternalServerError; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microso...

full failure pattern:

ERROR CODE: InternalServerError; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapError | 0xc000334400>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-57zxqg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31",
      "name": "3c3b1341-fda3-41e9-8dd4-5c30342a2d31",
      "status": "Failed",
      "startTime": "2026-03-30T06:12:39.328826001Z",
      "endTime": "2026-03-30T06:13:08.581005823Z",
      "error": {
        "code": "InternalServerError",
        "message": "Failed to get virtual network 'customer-vnet-name-4-19' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client '3820d679-b809-47f8-8f80-79cbd25d1d89' with object id '62b94c2a-37fd-4b0a-b97c-1370b69f6c0c' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-57zxqg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31\",\n  \"name\": \"3c3b1341-fda3-41e9-8dd4-5c30342a2d31\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:12:39.328826001Z\",\n  \"endTime\": \"2026-03-30T06:13:08.581005823Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-name-4-19' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client '3820d679-b809-47f8-8f80-79cbd25d1d89' with object id '62b94c2a-37fd-4b0a-b97c-1370b69f6c0c' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc001475530>{
            ErrorCode: "InternalServerError",
            StatusCode: 200,
            RawResponse: {
                Status: "200 OK",
                StatusCode: 200,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Pragma": ["no-cache"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/f58e194a-3f86-4008-98dc-aa5f60802195",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Msedge-Ref": [
                        "Ref A: 60DFA7F6143244D1924498FA54547EC7 Ref B: CHI261140510034 Ref C: 2026-03-30T06:13:10Z",
                    ],
                    "Date": [
                        "Mon, 30 Mar 2026 06:13:10 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Expires": ["-1"],
                    "X-Ms-Correlation-Request-Id": [
                        "bdec501d-0ca0-4486-8500-77afaa68e37a",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "CANADAEAST:20260330T061310Z:bdec501d-0ca0-4486-8500-77afaa68e37a",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["9"],
                    "Content-Length": ["1563"],
                    "Content-Type": ["application/json"],
                    "X-M...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapError | 0xc000334400>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-57zxqg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31",
      "name": "3c3b1341-fda3-41e9-8dd4-5c30342a2d31",
      "status": "Failed",
      "startTime": "2026-03-30T06:12:39.328826001Z",
      "endTime": "2026-03-30T06:13:08.581005823Z",
      "error": {
        "code": "InternalServerError",
        "message": "Failed to get virtual network 'customer-vnet-name-4-19' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client '3820d679-b809-47f8-8f80-79cbd25d1d89' with object id '62b94c2a-37fd-4b0a-b97c-1370b69f6c0c' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-57zxqg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31\",\n  \"name\": \"3c3b1341-fda3-41e9-8dd4-5c30342a2d31\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:12:39.328826001Z\",\n  \"endTime\": \"2026-03-30T06:13:08.581005823Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-name-4-19' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client '3820d679-b809-47f8-8f80-79cbd25d1d89' with object id '62b94c2a-37fd-4b0a-b97c-1370b69f6c0c' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc001475530>{
            ErrorCode: "InternalServerError",
            StatusCode: 200,
            RawResponse: {
                Status: "200 OK",
                StatusCode: 200,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Pragma": ["no-cache"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/f58e194a-3f86-4008-98dc-aa5f60802195",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Msedge-Ref": [
                        "Ref A: 60DFA7F6143244D1924498FA54547EC7 Ref B: CHI261140510034 Ref C: 2026-03-30T06:13:10Z",
                    ],
                    "Date": [
                        "Mon, 30 Mar 2026 06:13:10 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Expires": ["-1"],
                    "X-Ms-Correlation-Request-Id": [
                        "bdec501d-0ca0-4486-8500-77afaa68e37a",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "CANADAEAST:20260330T061310Z:bdec501d-0ca0-4486-8500-77afaa68e37a",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["9"],
                    "Content-Length": ["1563"],
                    "Content-Type": ["application/json"],
                    "X-M...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.19	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-03-30 06:09	n/a	2038498730092531712

ERROR CODE: InternalServerError; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microso...

full failure pattern:

ERROR CODE: InternalServerError; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-<id>' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: Unexpected error:
    <*fmt.wrapError | 0xc00131a180>: 
    failed to create HCP cluster cluster-zstream-4-21-xvw7fn: failed waiting for cluster="cluster-zstream-4-21-xvw7fn" in resourcegroup="rg-zstream-upgrade-4-21-qf98rq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23",
      "name": "66f6aa6d-0e58-4cf7-a817-132dc9b1db23",
      "status": "Failed",
      "startTime": "2026-04-02T01:34:57.397134314Z",
      "endTime": "2026-04-02T01:36:07.715905327Z",
      "error": {
        "code": "InternalServerError",
        "message": "Failed to get virtual network 'customer-vnet-zstream-xvw7fn' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client '0c358081-82b1-4f65-97bc-be8647a5b112' with object id 'b3c21201-bf1b-4718-b697-526dbc98e7e9' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-21-xvw7fn: failed waiting for cluster=\"cluster-zstream-4-21-xvw7fn\" in resourcegroup=\"rg-zstream-upgrade-4-21-qf98rq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23\",\n  \"name\": \"66f6aa6d-0e58-4cf7-a817-132dc9b1db23\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T01:34:57.397134314Z\",\n  \"endTime\": \"2026-04-02T01:36:07.715905327Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-zstream-xvw7fn' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client '0c358081-82b1-4f65-97bc-be8647a5b112' with object id 'b3c21201-bf1b-4718-b697-526dbc98e7e9' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00131a160>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-21-xvw7fn\" in resourcegroup=\"rg-zstream-upgrade-4-21-qf98rq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23\",\n  \"name\": \"66f6aa6d-0e58-4cf7-a817-132dc9b1db23\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T01:34:57.397134314Z\",\n  \"endTime\": \"2026-04-02T01:36:07.715905327Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-zstream-xvw7fn' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: Unexpected error:
    <*fmt.wrapError | 0xc00131a180>: 
    failed to create HCP cluster cluster-zstream-4-21-xvw7fn: failed waiting for cluster="cluster-zstream-4-21-xvw7fn" in resourcegroup="rg-zstream-upgrade-4-21-qf98rq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23",
      "name": "66f6aa6d-0e58-4cf7-a817-132dc9b1db23",
      "status": "Failed",
      "startTime": "2026-04-02T01:34:57.397134314Z",
      "endTime": "2026-04-02T01:36:07.715905327Z",
      "error": {
        "code": "InternalServerError",
        "message": "Failed to get virtual network 'customer-vnet-zstream-xvw7fn' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client '0c358081-82b1-4f65-97bc-be8647a5b112' with object id 'b3c21201-bf1b-4718-b697-526dbc98e7e9' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-21-xvw7fn: failed waiting for cluster=\"cluster-zstream-4-21-xvw7fn\" in resourcegroup=\"rg-zstream-upgrade-4-21-qf98rq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23\",\n  \"name\": \"66f6aa6d-0e58-4cf7-a817-132dc9b1db23\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T01:34:57.397134314Z\",\n  \"endTime\": \"2026-04-02T01:36:07.715905327Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-zstream-xvw7fn' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client '0c358081-82b1-4f65-97bc-be8647a5b112' with object id 'b3c21201-bf1b-4718-b697-526dbc98e7e9' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00131a160>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-21-xvw7fn\" in resourcegroup=\"rg-zstream-upgrade-4-21-qf98rq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66f6aa6d-0e58-4cf7-a817-132dc9b1db23\",\n  \"name\": \"66f6aa6d-0e58-4cf7-a817-132dc9b1db23\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-04-02T01:34:57.397134314Z\",\n  \"endTime\": \"2026-04-02T01:36:07.715905327Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-zstream-xvw7fn' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qf98rq/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-xvw7fn\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-02 00:18	n/a	2039497738340536320

add password: error status code received from the API

full failure pattern:

add password: error status code received from the API

Signal:

e2e

DEV, INT, STG

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:130]: Unexpected error:
    <*fmt.wrapError | 0xc0008802e0>: 
    add password: error status code received from the API
    {
        msg: "add password: error status code received from the API",
        err: <*odataerrors.ODataError | 0xc000b91aa0>{
            ApiError: {
                Message: "",
                ResponseStatusCode: 404,
                ResponseHeaders: {
                    header: {
                        headers: {
                            "strict-transport-security": {
                                "max-age=31536000": {},
                            },
                            "client-request-id": {
                                "609e61cb-e058-40bf-954c-a040be046529": {},
                            },
                            "content-type": {
                                "application/json": {},
                            },
                            "x-ms-ags-diagnostic": {
                                "{\"ServerInfo\":{\"DataCenter\":\"East US\",\"Slice\":\"E\",\"Ring\":\"5\",\"ScaleUnit\":\"001\",\"RoleInstance\":\"MN1PEPF00010855\"}}": {},
                            },
                            "x-ms-resource-unit": {"1": {}},
                            "date": {
                                "Tue, 31 Mar 2026 03:00:12 GMT": {},
                            },
                            "vary": {
                                "Accept-Encoding": {},
                            },
                            "request-id": {
                                "609e61cb-e058-40bf-954c-a040be046529": {},
                            },
                            "cache-control": {"no-cache": {}},
                        },
                    },
                },
            },
            backingStore: <*store.InMemoryBackingStore | 0xc000880280>{
                returnOnlyChangedValues: false,
                initializationCompleted: false,
                store: {
                    "errorEscaped": <*odataerrors.MainError | 0xc0014a1650>{
                        backingStore: <*store.InMemoryBackingStore | 0xc0008802a0>{
                            returnOnlyChangedValues: false,
                            initializationCompleted: false,
                            store: {
                                "code": <*string | 0xc0014a14d0>"Request_ResourceNotFound",
                                "message": <*string | 0xc0014a1520>"Resource 'b5d9870c-3832-4af0-9062-5ced2786c558' does not exist or one of its queried reference-property objects are not present.",
                                "innerError": <*odataerrors.InnerError | 0xc0014a16b0>{
                                    backingStore: <*store.InMemoryBackingStore | 0xc0008802c0>{
                                        returnOnlyChangedValues: false,
                                        initializationCompleted: false,
                                        store: {...: ..., ...: ...},
                                        subscribers: {},
                                        changedValues: {...: ..., ...: ...},
                                    },
                                },
                            },
                            subscribers: {},
                            changedValues: {"code": false, "message": false, "innerError": false},
                        },
                    },
                },
                subscribers: {},
                changedValues: {
                    "errorEscaped": false,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:130]: Unexpected error:
    <*fmt.wrapError | 0xc0008802e0>: 
    add password: error status code received from the API
    {
        msg: "add password: error status code received from the API",
        err: <*odataerrors.ODataError | 0xc000b91aa0>{
            ApiError: {
                Message: "",
                ResponseStatusCode: 404,
                ResponseHeaders: {
                    header: {
                        headers: {
                            "strict-transport-security": {
                                "max-age=31536000": {},
                            },
                            "client-request-id": {
                                "609e61cb-e058-40bf-954c-a040be046529": {},
                            },
                            "content-type": {
                                "application/json": {},
                            },
                            "x-ms-ags-diagnostic": {
                                "{\"ServerInfo\":{\"DataCenter\":\"East US\",\"Slice\":\"E\",\"Ring\":\"5\",\"ScaleUnit\":\"001\",\"RoleInstance\":\"MN1PEPF00010855\"}}": {},
                            },
                            "x-ms-resource-unit": {"1": {}},
                            "date": {
                                "Tue, 31 Mar 2026 03:00:12 GMT": {},
                            },
                            "vary": {
                                "Accept-Encoding": {},
                            },
                            "request-id": {
                                "609e61cb-e058-40bf-954c-a040be046529": {},
                            },
                            "cache-control": {"no-cache": {}},
                        },
                    },
                },
            },
            backingStore: <*store.InMemoryBackingStore | 0xc000880280>{
                returnOnlyChangedValues: false,
                initializationCompleted: false,
                store: {
                    "errorEscaped": <*odataerrors.MainError | 0xc0014a1650>{
                        backingStore: <*store.InMemoryBackingStore | 0xc0008802a0>{
                            returnOnlyChangedValues: false,
                            initializationCompleted: false,
                            store: {
                                "code": <*string | 0xc0014a14d0>"Request_ResourceNotFound",
                                "message": <*string | 0xc0014a1520>"Resource 'b5d9870c-3832-4af0-9062-5ced2786c558' does not exist or one of its queried reference-property objects are not present.",
                                "innerError": <*odataerrors.InnerError | 0xc0014a16b0>{
                                    backingStore: <*store.InMemoryBackingStore | 0xc0008802c0>{
                                        returnOnlyChangedValues: false,
                                        initializationCompleted: false,
                                        store: {...: ..., ...: ...},
                                        subscribers: {},
                                        changedValues: {...: ..., ...: ...},
                                    },
                                },
                            },
                            subscribers: {},
                            changedValues: {"code": false, "message": false, "innerError": false},
                        },
                    },
                },
                subscribers: {},
                changedValues: {
                    "errorEscaped": false,
                },
            },
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-03-31 02:00	n/a	2038798596630908928

failed creating deployment "test-vm" in resourcegroup="<resource-group>": PUT <url>

full failure pattern:

failed creating deployment "test-vm" in resourcegroup="<resource-group>": PUT <url>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:114]: Timed out after 1800.001s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc0012c8180>: 
    failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-s5czh5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-s5czh5/providers/Microsoft.Resources/deployments/test-vm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidTemplateDeployment
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidTemplateDeployment",
        "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.",
        "details": [
          {
            "code": "ResourceCountExceedsLimitDueToTemplate",
            "message": "Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress",
            "details": []
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed creating deployment \"test-vm\" in resourcegroup=\"e2e-cidr-connectivity-s5czh5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-s5czh5/providers/Microsoft.Resources/deployments/test-vm\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplateDeployment\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidTemplateDeployment\",\n    \"message\": \"The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.\",\n    \"details\": [\n      {\n        \"code\": \"ResourceCountExceedsLimitDueToTemplate\",\n        \"message\": \"Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress\",\n        \"details\": []\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000f127e0>{
            ErrorCode: "InvalidTemplateDeployment",
            StatusCode: 400,
            RawResponse: {
                Status: "400 Bad Request",
                StatusCode: 400,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["198"],
                    "X-Ms-Failure-Cause": ["gateway"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2998"],
                    "X-Ms-Request-Id": [
                        "cf5d91a3-a913-4d5b-9164-bccf81eb8eee",
                    ],
                    "X-Ms-Correlation-Request-Id": [
                        "cf5d91a3-a913-4d5b-9164-bccf81eb8eee",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "EASTUS2EUAP:20260330T065415Z:cf5d91a3-a913-4d5b-9164-bccf81eb8eee",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "Date": [
                        "Mon, 30 Mar 2026 06:54:14 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Pragma": ["no-cache"],
                    "Expires": ["-1"],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Msedge-Ref": [
                        "Ref A: FED97ACCAC0E49C1AB5718A871C140C7 Ref B: CHI261140514023 Ref C: 2026-03-30T06:54:14Z",
                    ],
                    "Content-Length": ["655"],
                    "Content-Type": [
                        "application/json; charset=utf-8",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0012c8040>{
                    s: "{\"error\":{\"code\":\"InvalidTemplateDeployment\",\"message\":\"The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.\",\"details\":[{\"code\":\"ResourceCountExceedsLimitDueToTemplate\",\"message\":\"Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress\",\"detai...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:114]: Timed out after 1800.001s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc0012c8180>: 
    failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-s5czh5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-s5czh5/providers/Microsoft.Resources/deployments/test-vm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidTemplateDeployment
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidTemplateDeployment",
        "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.",
        "details": [
          {
            "code": "ResourceCountExceedsLimitDueToTemplate",
            "message": "Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress",
            "details": []
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed creating deployment \"test-vm\" in resourcegroup=\"e2e-cidr-connectivity-s5czh5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-s5czh5/providers/Microsoft.Resources/deployments/test-vm\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplateDeployment\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidTemplateDeployment\",\n    \"message\": \"The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.\",\n    \"details\": [\n      {\n        \"code\": \"ResourceCountExceedsLimitDueToTemplate\",\n        \"message\": \"Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress\",\n        \"details\": []\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000f127e0>{
            ErrorCode: "InvalidTemplateDeployment",
            StatusCode: 400,
            RawResponse: {
                Status: "400 Bad Request",
                StatusCode: 400,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["198"],
                    "X-Ms-Failure-Cause": ["gateway"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2998"],
                    "X-Ms-Request-Id": [
                        "cf5d91a3-a913-4d5b-9164-bccf81eb8eee",
                    ],
                    "X-Ms-Correlation-Request-Id": [
                        "cf5d91a3-a913-4d5b-9164-bccf81eb8eee",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "EASTUS2EUAP:20260330T065415Z:cf5d91a3-a913-4d5b-9164-bccf81eb8eee",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "Date": [
                        "Mon, 30 Mar 2026 06:54:14 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Pragma": ["no-cache"],
                    "Expires": ["-1"],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Msedge-Ref": [
                        "Ref A: FED97ACCAC0E49C1AB5718A871C140C7 Ref B: CHI261140514023 Ref C: 2026-03-30T06:54:14Z",
                    ],
                    "Content-Length": ["655"],
                    "Content-Type": [
                        "application/json; charset=utf-8",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0012c8040>{
                    s: "{\"error\":{\"code\":\"InvalidTemplateDeployment\",\"message\":\"The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.\",\"details\":[{\"code\":\"ResourceCountExceedsLimitDueToTemplate\",\"message\":\"Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress\",\"detai...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-03-30 06:09	n/a	2038498730092531712

failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded

full failure pattern:

failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded

Signal:

e2e

DEV, INT

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:186]: Unexpected error:
    <*fmt.wrapError | 0xc0020085c0>: 
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="e2e-cilium-gpnkb5" to finish creating: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"cilium-np\" for cluster \"cilium-cluster\" in resourcegroup=\"e2e-cilium-gpnkb5\" to finish creating: context deadline exceeded",
        err: <context.deadlineExceededError>{},
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:186]: Unexpected error:
    <*fmt.wrapError | 0xc0020085c0>: 
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="e2e-cilium-gpnkb5" to finish creating: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"cilium-np\" for cluster \"cilium-cluster\" in resourcegroup=\"e2e-cilium-gpnkb5\" to finish creating: context deadline exceeded",
        err: <context.deadlineExceededError>{},
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-05 02:00	n/a	2040610572545822720