CIHealth
<
Single Day: Mar 17▾
Last 7 Days Weekly: Mar 16 - Mar 22 Sprint: Mar 16 - Mar 29 Single Day: Mar 17
Custom time window
> Reset
View JSON API

Environment: INT

Failure patterns: 4 · Failures matched: 7 · Total runs: 11 · Runs affected: 6

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Value for field 'CommonName' failed to satisfy matcher.
full failure pattern:
Value for field 'CommonName' failed to satisfy matcher.
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e436.36%4 of 11 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsMar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 · Mar 16: 3 · Mar 17: 4none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:98]: expect certificate to be issued by Microsoft
Value for field 'CommonName' failed to satisfy matcher.
Expected
    <string>: Microsoft TLS G2 RSA CA OCSP 02
to match regular expression
    <string>: Microsoft\sAzure\sRSA\sTLS\sIssuing\sCA\s[0-9]+
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:98]: expect certificate to be issued by Microsoft
Value for field 'CommonName' failed to satisfy matcher.
Expected
    <string>: Microsoft TLS G2 RSA CA OCSP 02
to match regular expression
    <string>: Microsoft\sAzure\sRSA\sTLS\sIssuing\sCA\s[0-9]+
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:98]: expect certificate to be issued by Microsoft
Value for field 'CommonName' failed to satisfy matcher.
Expected
    <string>: Microsoft TLS G2 RSA CA OCSP 04
to match regular expression
    <string>: Microsoft\sAzure\sRSA\sTLS\sIssuing\sCA\s[0-9]+
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:98]: expect certificate to be issued by Microsoft
Value for field 'CommonName' failed to satisfy matcher.
Expected
    <string>: Microsoft TLS G2 RSA CA OCSP 04
to match regular expression
    <string>: Microsoft\sAzure\sRSA\sTLS\sIssuing\sCA\s[0-9]+
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates4
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2033800140292100096
n/a2033776026961580032
n/a2033745294096601088
n/a2033717274690457600
ERROR CODE: Conflict; detail message Cannot request credential while credentials are being revoked; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: Conflict; detail message Cannot request credential while credentials are being revoked; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e19.09%1 of 11 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsMar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 · Mar 16: 1 · Mar 17: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:247]: Unexpected error:
    <*fmt.wrapError | 0xc00108e320>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-864pbb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-j5ldzb/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while credentials are being revoked",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-864pbb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-j5ldzb"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-864pbb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-j5ldzb/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot request credential while credentials are being revoked\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-864pbb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-j5ldzb\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000b9e4b0>{
            ErrorCode: "Conflict",
            StatusCode: 409,
            RawResponse: {
                Status: "409 Conflict",
                StatusCode: 409,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Content-Length": ["347"],
                    "X-Ms-Request-Id": [
                        "fc853af8-ce3e-44b9-912c-d8437b3a92df",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Msedge-Ref": [
                        "Ref A: ABDD73787FD640E9A40CC3967DD4D82A Ref B: CHI261140511062 Ref C: 2026-03-17T19:05:56Z",
                    ],
                    "Retry-After": ["10"],
                    "X-Envoy-Upstream-Service-Time": ["41"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Correlation-Request-Id": [
                        "816f9dcd-9515-4918-8720-1c04aea5b13f",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Cache-Control": ["no-cache"],
                    "Pragma": ["no-cache"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/canadacentral/195f8080-e1de-4af7-9ab6-48062a2f14a1",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "CANADACENTRAL:20260317T190557Z:816f9dcd-9515-4918-8720-1c04aea5b13f",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "Content-Type": ["application/json"],
                    "Expires": ["-1"],
                    "X-Ms-Error-Code": ["Conflict"],
                    "Date": [
                        "Tue, 17 Mar 2026 19:05:57 GMT",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc00108e200>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 115, 32, 97, 114, 101, 32, 98, 101, 105, 110, 103, 32, 114, 101, 118, 111, 107, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 52, 102, 48, 54, 49, 57, 102, 45, 1...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:247]: Unexpected error:
    <*fmt.wrapError | 0xc00108e320>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-864pbb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-j5ldzb/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while credentials are being revoked",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-864pbb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-j5ldzb"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-864pbb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-j5ldzb/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot request credential while credentials are being revoked\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-864pbb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-j5ldzb\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000b9e4b0>{
            ErrorCode: "Conflict",
            StatusCode: 409,
            RawResponse: {
                Status: "409 Conflict",
                StatusCode: 409,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Content-Length": ["347"],
                    "X-Ms-Request-Id": [
                        "fc853af8-ce3e-44b9-912c-d8437b3a92df",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Msedge-Ref": [
                        "Ref A: ABDD73787FD640E9A40CC3967DD4D82A Ref B: CHI261140511062 Ref C: 2026-03-17T19:05:56Z",
                    ],
                    "Retry-After": ["10"],
                    "X-Envoy-Upstream-Service-Time": ["41"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Correlation-Request-Id": [
                        "816f9dcd-9515-4918-8720-1c04aea5b13f",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Cache-Control": ["no-cache"],
                    "Pragma": ["no-cache"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/canadacentral/195f8080-e1de-4af7-9ab6-48062a2f14a1",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "CANADACENTRAL:20260317T190557Z:816f9dcd-9515-4918-8720-1c04aea5b13f",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "Content-Type": ["application/json"],
                    "Expires": ["-1"],
                    "X-Ms-Error-Code": ["Conflict"],
                    "Date": [
                        "Tue, 17 Mar 2026 19:05:57 GMT",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc00108e200>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 67, 111, 110, 102, 108, 105, 99, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 97, 110, 110, 111, 116, 32, 114, 101, 113, 117, 101, 115, 116, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 32, 119, 104, 105, 108, 101, 32, 99, 114, 101, 100, 101, 110, 116, 105, 97, 108, 115, 32, 97, 114, 101, 32, 98, 101, 105, 110, 103, 32, 114, 101, 118, 111, 107, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 52, 102, 48, 54, 49, 57, 102, 45, 1...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2033974052363702272
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
full failure pattern:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e19.09%1 of 11 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsMar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 · Mar 16: 0 · Mar 17: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:254]: New admin credentials should work after revocation
Expected success, but got an error:
    <*errors.joinError | 0xc000d20db0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0xc000d37080>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0xc000d37060>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0xc000c62be0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xc000d83500>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0xc000d83360>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0xc0009d0280>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:254]: New admin credentials should work after revocation
Expected success, but got an error:
    <*errors.joinError | 0xc000d20db0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0xc000d37080>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0xc000d37060>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0xc000c62be0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xc000d83500>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0xc000d83360>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0xc0009d0280>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2034017373396144128
found 1 managed resource groups left behind HCP clusters in cluster-listing-2s9rvc
full failure pattern:
found 1 managed resource groups left behind HCP clusters in cluster-listing-2s9rvc
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e19.09%1 of 11 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsMar 11: 0 · Mar 12: 0 · Mar 13: 0 · Mar 14: 0 · Mar 15: 0 · Mar 16: 0 · Mar 17: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
    <*errors.joinError | 0xc00111f080>: 
    found 1 managed resource groups left behind HCP clusters in cluster-listing-2s9rvc
    {
        errs: [
            <*errors.errorString | 0xc00135d4d0>{
                s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-2s9rvc",
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
    <*errors.joinError | 0xc00111f080>: 
    found 1 managed resource groups left behind HCP clusters in cluster-listing-2s9rvc
    {
        errs: [
            <*errors.errorString | 0xc00135d4d0>{
                s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-2s9rvc",
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to list HCP clusters without node pools at both subscription and resource group levels1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2033776026961580032