elastic-agent uninstall can panic

[cmacknz]

• Relates [Self-Managed][Windows]: unexpected return pc for runtime.deferreturn error while running uninstall command for windows fleet-server. fleet-server#2935
• Relates Windows uninstall can fail with Error: failed to remove installation directory or Access is denied #3342

Windows 11 console when uninstalling an agent built from https://github.com/elastic/elastic-agent/commits/2ff5f632e4a1ffe02ff78ca7fc7b17f95f8e310a

"C:\Program Files\Elastic\Agent\elastic-agent.exe" uninstall
Elastic Agent will be uninstalled from your system at C:\Program Files\Elastic\Agent. Do you want to continue? [Y/n]:Y
Stopping service... DONE
Stopping upgrade watcher; none found... DONE
Uninstalling service component endpoint............................................................................................ DONE
Removing service... DONE
Removing install directory...................unexpected fault address 0x1f86480
fatal error: fault
[signal 0xc0000005 code=0x1 addr=0x1f86480 pc=0x639c20]

goroutine 6 [running]:
fatal error: unexpected signal during runtime execution
panic during panic
[signal 0xc0000005 code=0x1 addr=0x63a425 pc=0x660b40]

runtime stack:
fatal error: unexpected signal during runtime execution
stack trace unavailable

After this happened almost everything was removed, just one log has left in
"C:\Program Files\Elastic\Agent\elastic-agent-20231103-5.ndjson" which is below:

{"log.level":"error","@timestamp":"2023-11-03T17:23:35.588+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:35: info: Main.cpp:299 Executing uninstall","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:35.589+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:35: info: Internal.cpp:51 Found config path [C:\\Program Files\\Elastic\\Endpoint\\elastic-endpoint.yaml]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:35.590+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:35: debug: Utilities.cpp:420 Tamper protection disabled","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:35.590+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:35: info: InstallLib.cpp:1153 Skipping uninstall token validation as tamper protection is not enabled.","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:35.590+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:35: debug: Service.cpp:804 PPL is supported. This process is unprotected. (TrustLevelSid: absent)","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:35.590+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:35: info: Util.cpp:795 Sending service command to facilitate uninstall","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:35.591+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:35: info: Util.cpp:822 Service command to facilitate uninstall succeeded","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.294+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\elastic-endpoint.yaml]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.294+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\WINDOWS\\System32\\Drivers\\elastic-endpoint-driver.sys]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.305+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\SecurityProductInformation.ini]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.305+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\elastic-endpoint.exe]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.308+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\endpointpe-v4-model]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.308+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\endpointpe-v4-exceptionlist]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.308+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\endpointpe-v4-blocklist]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.308+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\production-ransomware-v1-windows]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.308+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\diagnostic-ransomware-v1-windows]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.308+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\global-exceptionlist-windows]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.309+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\global-trustlist-windows-v1]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.309+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\global-eventfilterlist-windows-v1]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.309+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\global-configuration-v1]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.309+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\tamper-protection-config-v1]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.309+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\diagnostic-rules-windows-v1]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.309+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\production-rules-windows-v1]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.309+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\manifest.json]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.309+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\manifest.sig]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.310+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\production-malware-signature-v1-windows]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.310+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\diagnostic-malware-signature-v1-windows]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.310+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\diagnostic-endpointpe-v4-exceptionlist]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.310+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\diagnostic-endpointpe-v4-blocklist]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.310+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\state\\artifacts\\global-artifacts\\diagnostic-endpointpe-v4-model]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.310+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\cache\\resources\\elastic-endpoint-security.png]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.310+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\LICENSE.txt]","context":"command output","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-03T17:23:59.311+0100","log.origin":{"file.name":"runtime/service_command.go","file.line":69},"message":"2023-11-03 16:23:59: debug: File.cpp:453 Removing [C:\\Program Files\\Elastic\\Endpoint\\NOTICE.txt]","context":"command output","ecs.version":"1.6.0"}

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

The single log file being left behind is likely because of #3342

[cmacknz]

I have a copy of agent BC7 on Windows downloaded which is the build this was reproduced on, I'd upload it but it is apparently too large.

I hope we can use https://pkg.go.dev/debug/gosym#Table.PCToLine to map the PC address here to a line in the source.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

Closing, not reproducible.