feat(inputs.nvidia-smi): Add probe_on_startup option

[LandonTClipp]

Summary

There are some cases where the nvidia-smi plugin might be found in PATH and executable, but upon running it might always return a non-zero exit code. For various reasons, in the environment I work in, this might be expected. It's thus disruptive for system logs to be polluted with infinite error messages. It's preferable in this situation to check if nvidia-smi returns a good result on plugin startup, and if not, allow the error to be bubbled up and handled according to startup_error_behavior.

Checklist

• No AI generated code was used in this PR

Related issues

resolves #15915

[srebhan]

Why not returning a startup error without adding a new option?

[LandonTClipp]

Why not returning a startup error without adding a new option?

Backwards compatibility reasons, I'm not sure if someone's system might strangely rely on this old behavior. I could envision a situation where nvidia-smi fails when called but if someone did not set startup_error_behavior = "ignore" then this would suddenly make their whole service shutdown (I think). There is also a potential issue with the plugin startup getting hung if nvidia-smi itself hangs. I'm just being conservative by retaining the old behavior.

What do you think?

[srebhan]

The question is, would an error in executing the command be repairable at runtime? If that's not the case, I'd say go without the option as people still do have a way to fix the situation using another startup-error-behavior. Otherwise, we should add this option...

[LandonTClipp]

The question is, would an error in executing the command be repairable at runtime?

I think it depends on the issue at hand. In some cases it could be reparable, like if hot-rebooting a GPU fixes device memory corruption. In other situations like mine, it's not.

Personally, I think there is no risk at all to adding this option to at least give people the choice. Not adding the option has some risk, so why not do the zero risk option?

[LandonTClipp]

@srebhan any thoughts on the above?

PS thanks for the responsiveness thus far 🙏🏻

[srebhan]

@LandonTClipp, adding options that are not necessary complicates things, making it more difficult for the user to find out what to do and sometimes even cause ambiguities. Anyway, I think you do have a point in adding the option so let's go with this.

[LandonTClipp]

@srebhan all comments addressed.

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

[srebhan]

Awesome work @LandonTClipp! Thanks!

[LandonTClipp]

@srebhan I wonder if we should revert this PR in lieu of the upcoming probe option we're spec-ing out? You haven't gone through another release cycle yet, so now would be our opportunity.