CIHealth
<
Single Day: Apr 6▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
Interrupted by User
22:17:03 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
timeout during CreateHCPClusterAndWait; context deadline exceeded
21:58:34 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014aaa80>: 
    failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-2trfm7, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-2trfm7" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-2trfm7, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-2trfm7, error: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-2trfm7\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-2trfm7, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000ef8f30>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-2trfm7",
            },
            <*fmt.wrapErrors | 0xc0014aaa50>{
                msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-2trfm7\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-2trfm7, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000ef8f30>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-2trfm7",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014aaa80>: 
    failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-2trfm7, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-2trfm7" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-2trfm7, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-2trfm7, error: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-2trfm7\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-2trfm7, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000ef8f30>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-2trfm7",
            },
            <*fmt.wrapErrors | 0xc0014aaa50>{
                msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-2trfm7\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-2trfm7, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000ef8f30>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-2trfm7",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Interrupted by User
21:58:34 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
Interrupted by User
21:47:22 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e, unknown failed n/a 4
e2e (3)
Interrupted by User
18:04:54 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
ERROR CODE: Conflict; detail message Cannot update resource while resource is "accepted"; provider Microsoft.RedHatOpenShift
18:04:54 UTC · failed at e2e · test Customer should be able to list HCP clusters without node pools at the resource group level · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:85]: Unexpected error:
    <*fmt.wrapError | 0xc0001769a0>: 
    failed to create HCP cluster list-test-cluster-p4ct5p: failed starting cluster creation "list-test-cluster-p4ct5p" in resourcegroup="cluster-listing-m5pqxl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"accepted\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster list-test-cluster-p4ct5p: failed starting cluster creation \"list-test-cluster-p4ct5p\" in resourcegroup=\"cluster-listing-m5pqxl\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot update resource while resource is \\\"accepted\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000176980>{
            msg: "failed starting cluster creation \"list-test-cluster-p4ct5p\" in resourcegroup=\"cluster-listing-m5pqxl\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot update resource while resource is \\\"accepted\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000c6f590>{
                ErrorCode: "Conflict",
                StatusCode: 409,
                RawResponse: {
                    Status: "409 Conflict",
                    StatusCode: 409,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Date": [
                            "Mon, 06 Apr 2026 18:28:48 GMT",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/f8dcff39-92dd-4b53-bfd5-a54e320e9c99",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260406T182849Z:c7e70c50-1876-4215-b20d-f5fe6a8b0f35",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Msedge-Ref": [
                            "Ref A: 6C605D2072634CEB987C0EC2A7E6913D Ref B: CH1AA2020603011 Ref C: 2026-04-06T18:28:45Z",
                        ],
                        "Cache-Control": ["no-cache"],
                        "X-Ms-Error-Code": ["Conflict"],
                        "X-Ms-Request-Id": [
                            "057ea031-c736-4cfd-82cb-048ffa7ce574",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Ms-Correlation-Request-Id": [
                            "c7e70c50-18...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:85]: Unexpected error:
    <*fmt.wrapError | 0xc0001769a0>: 
    failed to create HCP cluster list-test-cluster-p4ct5p: failed starting cluster creation "list-test-cluster-p4ct5p" in resourcegroup="cluster-listing-m5pqxl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"accepted\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster list-test-cluster-p4ct5p: failed starting cluster creation \"list-test-cluster-p4ct5p\" in resourcegroup=\"cluster-listing-m5pqxl\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot update resource while resource is \\\"accepted\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000176980>{
            msg: "failed starting cluster creation \"list-test-cluster-p4ct5p\" in resourcegroup=\"cluster-listing-m5pqxl\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot update resource while resource is \\\"accepted\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000c6f590>{
                ErrorCode: "Conflict",
                StatusCode: 409,
                RawResponse: {
                    Status: "409 Conflict",
                    StatusCode: 409,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Date": [
                            "Mon, 06 Apr 2026 18:28:48 GMT",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/f8dcff39-92dd-4b53-bfd5-a54e320e9c99",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260406T182849Z:c7e70c50-1876-4215-b20d-f5fe6a8b0f35",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Msedge-Ref": [
                            "Ref A: 6C605D2072634CEB987C0EC2A7E6913D Ref B: CH1AA2020603011 Ref C: 2026-04-06T18:28:45Z",
                        ],
                        "Cache-Control": ["no-cache"],
                        "X-Ms-Error-Code": ["Conflict"],
                        "X-Ms-Request-Id": [
                            "057ea031-c736-4cfd-82cb-048ffa7ce574",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Ms-Correlation-Request-Id": [
                            "c7e70c50-18...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
Deserializaion Error: no output from command
18:04:54 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel
Show raw failure
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775503018 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:237
    STEP: creating resource group @ 04/06/26 19:16:58.68
  "ts"="2026-04-06 19:16:58.680121" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-rrxpd6-9pwrvq"
    STEP: creating cluster parameters at control plane version @ 04/06/26 19:17:02.655
    STEP: creating customer resources @ 04/06/26 19:17:02.751
    STEP: creating the HCP cluster with version 4.20.17 @ 04/06/26 19:18:03.735
  "ts"="2026-04-06 19:18:03.735663" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-rrxpd6" "resourceGroup"="rg-np-version-upgrade-rrxpd6-9pwrvq"
    STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 04/06/26 19:27:58.589
    STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/06/26 19:37:32.137
    STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/06/26 19:39:39.317
    STEP: capturing node release images before upgrade @ 04/06/26 19:39:39.492
    STEP: triggering nodepool upgrade to version 4.20.17 @ 04/06/26 19:39:39.666
    STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/06/26 19:40:43.947
    STEP: verifying node pool GET still reflects the new version @ 04/06/26 20:08:51.474
  "ts"="2026-04-06 20:08:51.771658" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-06 20:08:51.771800" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-rrxpd6-zbqc7q" "resourceGroup"="rg-np-version-upgrade-rrxpd6-9pwrvq"
  "ts"="2026-04-06 20:08:51.932025" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-rrxpd6-zbqc7q" "resourceGroup"=""
  "ts"="2026-04-06 20:08:53.805410" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz"
  "ts"="2026-04-06 20:08:53.805517" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz"
  "ts"="2026-04-06 20:08:53.805583" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-06 20:08:53.807054" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-134"
  "ts"="2026-04-06 20:08:53.807062" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-rrxpd6-9pwrvq"
  "ts"="2026-04-06 20:08:53.990241" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-rrxpd6-9pwrvq" "deployment"="pooledNonMsiScopedAssignments-whdkgg53trd2y"
  "ts"="2026-04-06 20:08:54.131102" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-134" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-06 20:08:54.324615" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-134" "deployment"="msi-deployment-134"
  "ts"="2026-04-06 20:08:54.518036" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-rrxpd6-9pwrvq" "deployment"="customer-infra-np-version-upgrade-cluster-rrxpd6-zbqc7q"
  "ts"="2026-04-06 20:08:54.673903" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-06 20:08:54.674011" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-06 20:08:54.674156" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-rrxpd6-9pwrvq"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775503018 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:237
    STEP: creating resource group @ 04/06/26 19:16:58.68
  "ts"="2026-04-06 19:16:58.680121" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-rrxpd6-9pwrvq"
    STEP: creating cluster parameters at control plane version @ 04/06/26 19:17:02.655
    STEP: creating customer resources @ 04/06/26 19:17:02.751
    STEP: creating the HCP cluster with version 4.20.17 @ 04/06/26 19:18:03.735
  "ts"="2026-04-06 19:18:03.735663" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-rrxpd6" "resourceGroup"="rg-np-version-upgrade-rrxpd6-9pwrvq"
    STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 04/06/26 19:27:58.589
    STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/06/26 19:37:32.137
    STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/06/26 19:39:39.317
    STEP: capturing node release images before upgrade @ 04/06/26 19:39:39.492
    STEP: triggering nodepool upgrade to version 4.20.17 @ 04/06/26 19:39:39.666
    STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/06/26 19:40:43.947
    STEP: verifying node pool GET still reflects the new version @ 04/06/26 20:08:51.474
  "ts"="2026-04-06 20:08:51.771658" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-06 20:08:51.771800" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-rrxpd6-zbqc7q" "resourceGroup"="rg-np-version-upgrade-rrxpd6-9pwrvq"
  "ts"="2026-04-06 20:08:51.932025" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-rrxpd6-zbqc7q" "resourceGroup"=""
  "ts"="2026-04-06 20:08:53.805410" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz"
  "ts"="2026-04-06 20:08:53.805517" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz"
  "ts"="2026-04-06 20:08:53.805583" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-06 20:08:53.807054" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-134"
  "ts"="2026-04-06 20:08:53.807062" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-rrxpd6-9pwrvq"
  "ts"="2026-04-06 20:08:53.990241" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-rrxpd6-9pwrvq" "deployment"="pooledNonMsiScopedAssignments-whdkgg53trd2y"
  "ts"="2026-04-06 20:08:54.131102" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-134" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-06 20:08:54.324615" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-134" "deployment"="msi-deployment-134"
  "ts"="2026-04-06 20:08:54.518036" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-rrxpd6-9pwrvq" "deployment"="customer-infra-np-version-upgrade-cluster-rrxpd6-zbqc7q"
  "ts"="2026-04-06 20:08:54.673903" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-06 20:08:54.674011" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-06 20:08:54.674156" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-rrxpd6-9pwrvq"
Command Error: signal: killed
Deserializaion Error: no output from command
unknown (1)
Process did not finish before 2h0m0s timeout
18:04:54 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit
Show raw failure
{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-06T20:27:11Z"}
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 4
e2e (4)
Interrupted by User
08:46:03 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift
08:46:03 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:181]: Unexpected error:
    <*fmt.wrapError | 0xc000b98320>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-nxrhjt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-nxrhjt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000271d40>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Request-Id": [
                        "1041c110-64e3-4647-b077-e678f611e5d2",
                    ],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/4e5bcd42-e79e-41df-b439-262ce47043fb",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260406T100012Z:10c71c98-ea17-4248-afb0-40a2d688a221",
                    ],
                    "Pragma": ["no-cache"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Ms-Correlation-Request-Id": [
                        "10c71c98-ea17-4248-afb0-40a2d688a221",
                    ],
                    "Content-Length": ["107"],
                    "Expires": ["-1"],
                    "X-Envoy-Upstream-Service-Time": ["2235"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "X-Msedge-Ref": [
                        "Ref A: 587DD9200B6E4171A675EA5401F200EE Ref B: CH1AA2020603035 Ref C: 2026-04-06T10:00:10Z",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Content-Type": ["application/json"],
                    "Date": [
                        "Mon, 06 Apr 2026 10:00:12 GMT",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000b98220>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "managem...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:181]: Unexpected error:
    <*fmt.wrapError | 0xc000b98320>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-nxrhjt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-nxrhjt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000271d40>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Request-Id": [
                        "1041c110-64e3-4647-b077-e678f611e5d2",
                    ],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/4e5bcd42-e79e-41df-b439-262ce47043fb",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260406T100012Z:10c71c98-ea17-4248-afb0-40a2d688a221",
                    ],
                    "Pragma": ["no-cache"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Ms-Correlation-Request-Id": [
                        "10c71c98-ea17-4248-afb0-40a2d688a221",
                    ],
                    "Content-Length": ["107"],
                    "Expires": ["-1"],
                    "X-Envoy-Upstream-Service-Time": ["2235"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "X-Msedge-Ref": [
                        "Ref A: 587DD9200B6E4171A675EA5401F200EE Ref B: CH1AA2020603035 Ref C: 2026-04-06T10:00:10Z",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Content-Type": ["application/json"],
                    "Date": [
                        "Mon, 06 Apr 2026 10:00:12 GMT",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000b98220>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "managem...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
Interrupted by User
08:46:03 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
Interrupted by User
08:46:03 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
pod "<pod>" is forbidden: error fetching namespace "<namespace>": unable to find annotation openshift.io/sa.scc.uid-range
00:31:18 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:344]: Unexpected error:
    <*errors.StatusError | 0xc0012b7ae0>: 
    pods "wi-test-pod" is forbidden: error fetching namespace "e2e-oidc-wi": unable to find annotation openshift.io/sa.scc.uid-range
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "pods \"wi-test-pod\" is forbidden: error fetching namespace \"e2e-oidc-wi\": unable to find annotation openshift.io/sa.scc.uid-range",
            Reason: "Forbidden",
            Details: {Name: "wi-test-pod", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
            Code: 403,
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:344]: Unexpected error:
    <*errors.StatusError | 0xc0012b7ae0>: 
    pods "wi-test-pod" is forbidden: error fetching namespace "e2e-oidc-wi": unable to find annotation openshift.io/sa.scc.uid-range
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "pods \"wi-test-pod\" is forbidden: error fetching namespace \"e2e-oidc-wi\": unable to find annotation openshift.io/sa.scc.uid-range",
            Reason: "Forbidden",
            Details: {Name: "wi-test-pod", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
            Code: 403,
        },
    }
occurred