Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add UEFI Common Platform Error Record (CPER) support #846

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add UEFI Common Platform Error Record (CPER) support #846

wants to merge 1 commit into from

Conversation

wenjianhn
Copy link

CPER is the format used to describe platform hardware error by various tables, such as ERST, BERT and HEST etc.

The event severity message is printed here:
https://github.com/torvalds/linux/blob/v6.7/drivers/firmware/efi/cper.c#L639

Examples are as below.

Corrected error:
kernel: {37}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 162 kernel: {37}[Hardware Error]: It has been corrected by h/w and requires no further action kernel: {37}[Hardware Error]: event severity: corrected kernel: {37}[Hardware Error]: Error 0, type: corrected
kernel: {37}[Hardware Error]: section_type: memory error
kernel: {37}[Hardware Error]: error_status: 0x0000000000000400
kernel: {37}[Hardware Error]: physical_address: 0x000000b50c68ce80
kernel: {37}[Hardware Error]: node: 1 card: 4 module: 0 rank: 0 bank: 1 device: 14 row: 58165 column: 816
kernel: {37}[Hardware Error]: error_type: 2, single-bit ECC
kernel: {37}[Hardware Error]: DIMM location: CPU 2 DIMM 30

Recoverable error:
kernel: {3}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4 kernel: {3}[Hardware Error]: event severity: recoverable kernel: {3}[Hardware Error]: Error 0, type: recoverable kernel: {3}[Hardware Error]: fru_text: B1
kernel: {3}[Hardware Error]: section_type: memory error
kernel: {3}[Hardware Error]: error_status: 0x0000000000000400
kernel: {3}[Hardware Error]: physical_address: 0x000000393cfe5040
kernel: {3}[Hardware Error]: node: 2 card: 0 module: 0 rank: 0 bank: 3 device: 0 row: 34719 column: 320
kernel: {3}[Hardware Error]: DIMM location: not present. DMI handle: 0x0000

Fatal error:
kernel: BERT: Error records from previous boot:
kernel: [Hardware Error]: event severity: fatal
kernel: [Hardware Error]: Error 0, type: fatal
kernel: [Hardware Error]: fru_text: DIMM B5
kernel: [Hardware Error]: section_type: memory error
kernel: [Hardware Error]: error_status: 0x0000000000000400
kernel: [Hardware Error]: physical_address: 0x000000393d7e4040
kernel: [Hardware Error]: node: 2 card: 4 module: 0 rank: 0 bank: 3 device: 0 row: 34743 column: 256

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 11, 2024
@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 11, 2024
@k8s-ci-robot
Copy link
Contributor

Hi @wenjianhn. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Jan 11, 2024
@hakman
Copy link
Member

hakman commented Jan 18, 2024

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 18, 2024
@wenjianhn
Copy link
Author

/test pull-npd-e2e-test

@wenjianhn
Add UEFI Common Platform Error Record (CPER) support
ec831d2 
CPER is the format used to describe platform hardware error by various
tables, such as ERST, BERT and HEST etc.

The event severity message is printed here:
https://github.com/torvalds/linux/blob/v6.7/drivers/firmware/efi/cper.c#L639

Examples are as below.

Corrected error:
kernel: {37}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 162
kernel: {37}[Hardware Error]: It has been corrected by h/w and requires no further action
kernel: {37}[Hardware Error]: event severity: corrected
kernel: {37}[Hardware Error]:  Error 0, type: corrected
kernel: {37}[Hardware Error]:   section_type: memory error
kernel: {37}[Hardware Error]:   error_status: 0x0000000000000400
kernel: {37}[Hardware Error]:   physical_address: 0x000000b50c68ce80
kernel: {37}[Hardware Error]:   node: 1 card: 4 module: 0 rank: 0 bank: 1 device: 14 row: 58165 column: 816
kernel: {37}[Hardware Error]:   error_type: 2, single-bit ECC
kernel: {37}[Hardware Error]:   DIMM location: CPU 2 DIMM 30

Recoverable error:
kernel: {3}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4
kernel: {3}[Hardware Error]: event severity: recoverable
kernel: {3}[Hardware Error]:  Error 0, type: recoverable
kernel: {3}[Hardware Error]:  fru_text: B1
kernel: {3}[Hardware Error]:   section_type: memory error
kernel: {3}[Hardware Error]:   error_status: 0x0000000000000400
kernel: {3}[Hardware Error]:   physical_address: 0x000000393cfe5040
kernel: {3}[Hardware Error]:   node: 2 card: 0 module: 0 rank: 0 bank: 3 device: 0 row: 34719 column: 320
kernel: {3}[Hardware Error]:   DIMM location: not present. DMI handle: 0x0000

Fatal error:
kernel: BERT: Error records from previous boot:
kernel: [Hardware Error]: event severity: fatal
kernel: [Hardware Error]:  Error 0, type: fatal
kernel: [Hardware Error]:  fru_text: DIMM B5
kernel: [Hardware Error]:   section_type: memory error
kernel: [Hardware Error]:   error_status: 0x0000000000000400
kernel: [Hardware Error]:   physical_address: 0x000000393d7e4040
kernel: [Hardware Error]:   node: 2 card: 4 module: 0 rank: 0 bank: 3 device: 0 row: 34743 column: 256

Steps to test the new metrics.
# echo "{1}[Hardware Error]: event severity: fatal" >  /dev/kmsg
# echo "{1}[Hardware Error]: event severity: recoverable" >  /dev/kmsg
# echo "{1}[Hardware Error]: event severity: corrected" >  /dev/kmsg

Expected metrics are as below:
$ curl localhost:20257/metrics
# HELP problem_counter Number of times a specific type of problem have occurred.
# TYPE problem_counter counter
problem_counter{reason="HardwareErrorCorrected"} 1
problem_counter{reason="HardwareErrorFatal"} 1
problem_counter{reason="HardwareErrorRecoverable"} 1
# HELP problem_gauge Whether a specific type of problem is affecting the node or not.
# TYPE problem_gauge gauge
problem_gauge{reason="HardwareErrorFatal",type="HardwareErrorFatal"} 1

Signed-off-by: Jian Wen <[email protected]>
@hakman
Copy link
Member

hakman commented Jan 30, 2024

Thanks @wenjianhn!
/lgtm
/cc @vteratipally

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 30, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: hakman, wenjianhn
Once this PR has been reviewed and has the lgtm label, please assign xueweiz for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@wangzhen127
Copy link
Member

/retest

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Mark this PR as fresh with /remove-lifecycle stale
  • Close this PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 4, 2024
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Mark this PR as fresh with /remove-lifecycle rotten
  • Close this PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Aug 3, 2024
@hakman
Copy link
Member

hakman commented Aug 3, 2024

/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Aug 3, 2024
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 16, 2024
@k8s-ci-robot
Copy link
Contributor

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sjenning sjenning Awaiting requested review from sjenning

@wangzhen127 wangzhen127 Awaiting requested review from wangzhen127

@vteratipally vteratipally Awaiting requested review from vteratipally

Assignees

@hakman hakman

Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@wenjianhn @k8s-ci-robot @hakman @wangzhen127 @k8s-triage-robot