Support attach to application consoles (Vm and Container)

docs/DEBUGGING.md

@@ -93,6 +93,75 @@ tail -F /run/diag.out
 
 In addition this information is provided to application instances on the device using [the diag API endpoint](./ECO-METADATA.md).
 
+## Application console
+
+A running application on an EVE device has a console for input or output. You can attach to the application console from the EVE device as a control terminal if the application (VM or Container) listens to the TTY line and communicates with the virtual console /dev/hvc0 device. For example for popular linux distributions deployed as VM application this is usually the case.
+
+First list applications consoles of all running QEMU (KVM) processes:
+
+```bash
+# eve list-app-consoles
+PID     APP-UUID                                CONS-TYPE       CONS-ID
+---     --------                                ---------       ---------
+3883    e4e2f56d-b833-4562-a86f-be654d6387ba    VM              e4e2f56d-b833-4562-a86f-be654d6387ba.1.1/cons
+4072    f6d348cc-9c31-4f8b-8c4f-a4aae4590b97    CONTAINER       f6d348cc-9c31-4f8b-8c4f-a4aae4590b97.1.2/cons
+4072    f6d348cc-9c31-4f8b-8c4f-a4aae4590b97    VM              f6d348cc-9c31-4f8b-8c4f-a4aae4590b97.1.2/prime-cons
+
+```
+
+Where fields are:
+
+* PID       - the process ID of the QEMU process
+* APP-UUID  - UUID of the application
+* CONS-TYPE - Type of the console
+* CONS-ID   - ID of the console, should be used for attaching to the console by passing the console ID to the `eve attach-app-console` command
+
+Different application types may have different consoles (as mentioned above). An application of type "Virtual Machine" can only have a console of type "VM", which leads to the console of the user application; An application of the "Container" type has two types of console: the console of the "VM" type leads to the Virtual Machine that hosts the container, the console of the "CONTAINER" type leads to the user container itself.
+
+Choose console ID you need to attach and pass it as an argument to the `eve attach-app-console` command:
+
+```bash
+# eve attach-app-console e4e2f56d-b833-4562-a86f-be654d6387ba.1.1/cons
+[20:26:15.116] tio v1.37
+[20:26:15.116] Press ctrl-t q to quit
+[20:26:15.116] Connected
+<PRESS ENTER>
+
+Ubuntu 18.04.6 LTS user hvc0
+
+user login:
+```
+
+Note: `tio` utility is used as a simple TTY terminal, so in order to quit the session please press `ctrl-t q` or read the `tio` manual for additional commands.
+
+The same 'cons' console ID can be used for the Container application, but please be aware if container does not start a shell then terminal is very limited and can be used only for reading for the console output, but not for executing commands.
+
+In order to attach to the console of the hosting Vm of the Container application another console ID should be used which is named `prime-cons`:
+
+```bash
+# eve attach-app-console f6d348cc-9c31-4f8b-8c4f-a4aae4590b97.1.2/prime-cons
+[20:41:47.124] tio v1.37
+[20:41:47.124] Press ctrl-t q to quit
+[20:41:47.124] Connected
+<PRESS ENTER>
+~ #
+```
+
+The `prime-cons` console exists only for the Container applications and is always reachable for executing commands on the Vm which hosts corresponding container.
+
+Once terminal responds on the `prime-cons` console it is possible to enter container by executing the `eve-enter-container` command:
+
+```bash
+~ # eve-enter-container
+(none):/# ps awux
+PID   USER     TIME  COMMAND
+    1 root      0:00 /bin/sh
+    6 root      0:00 -ash
+    7 root      0:00 ps awux
+(none):/# exit
+~ #
+```
+
 ## Reboots
 
 EVE is architected in such a way that if any service is unresponsive for a period of time, the entire device will reboot. When this happens a BootReason is constructed and sent in the device info message to the controller. If there is a golang panic there can also be useful information found in `/persist/agentdebug/`.

pkg/alpine/mirrors/3.16/community

@@ -1,7 +1,22 @@
+fio
+fmt
+hwinfo
+i2c-tools-dev
+iw
+libgudev-dev
+librados
+librbd
+libvirt
+libvirt-client
+libvirt-common-drivers
+libvirt-daemon
+libvirt-libs
+libvirt-lxc
+libvirt-qemu
 libvncserver
 libvncserver-dev
-qemu-img
 perf
+pkgconf
 py3-cachecontrol
 py3-colorama
 py3-contextlib2
@@ -13,37 +28,23 @@ py3-msgpack
 py3-pep517
 py3-pip
 py3-progress
+py3-pycryptodome
 py3-pytoml
 py3-retrying
 py3-tomli
 py3-webencodings
-py3-pycryptodome
+qemu
+qemu-img
+qemu-system-x86_64
 sudo
-tini
-fio
 sysstat
-hwinfo
-iw
-i2c-tools-dev
-qemu-system-x86_64
-qemu
-tpm2-tss
+tini
+tio
 tpm2-abrmd
-tpm2-tss-esys
+tpm2-tss
 tpm2-tss-dev
-tpm2-tss-sys
-tpm2-tss-tctildr
+tpm2-tss-esys
 tpm2-tss-fapi
 tpm2-tss-rc
-pkgconf
-libgudev-dev
-librados
-librbd
-libvirt
-libvirt-client
-libvirt-common-drivers
-libvirt-daemon
-libvirt-libs
-libvirt-lxc
-libvirt-qemu
-fmt
+tpm2-tss-sys
+tpm2-tss-tctildr

pkg/debug/Dockerfile

@@ -4,7 +4,7 @@
 # has a fast path for stack unwinding. This also happens
 # to be a perfect place to put any other kind of debug info
 # into the package: see abuild/etc/abuild.conf.
-FROM lfedge/eve-alpine:fad44e3702708a8d044663a20fd98d933dddb41e as build
+FROM lfedge/eve-alpine:cbf02c2c126f210933ec9bdb142eb080b400fd76 as build
 ENV BUILD_PKGS abuild curl tar make linux-headers patch g++ git gcc ncurses-dev autoconf
 # Feel free to add additional packages here, but be aware that
 # EVE's rootfs image can be no larger than 300Mb (and don't
@@ -15,8 +15,8 @@ ENV PKGS openssl openssh-client openssh-server tini util-linux ca-certificates p
 
 # These packages are not available on the riscv arch, so I have no idea how
 # deliver those, but still install them on other archs.
-ENV PKGS_amd64 procps tar dmidecode iptables dhcpcd
-ENV PKGS_arm64 procps tar dmidecode iptables dhcpcd
+ENV PKGS_amd64 procps tar dmidecode iptables dhcpcd tio socat
+ENV PKGS_arm64 procps tar dmidecode iptables dhcpcd tio socat
 
 RUN eve-alpine-deploy.sh
 

pkg/dom0-ztools/rootfs/bin/eve

@@ -2,6 +2,7 @@
 
 CTR_CMD="ctr --namespace services.linuxkit"
 CTR_UA_CMD="ctr --namespace eve-user-apps"
+KVM_RUN_DIR="/run/hypervisor/kvm"
 
 help() {
   cat <<__EOT__
@@ -15,6 +16,8 @@ Welcome to EVE!
             pause <service>
             resume <service>
             destroy <service>
+            list-app-consoles   Outputs list of available consoles for attach
+            attach-app-console <console id>   Attach to the application console, see 'eve list-app-consoles' for details
             persist list
             persist attach <disk>
             config mount <mountpoint>
@@ -114,6 +117,10 @@ dump_mem() {
     http_debug_request "/dump/memory"
 }
 
+is_in_debug_service() {
+    grep -q '/eve/services/debug' < /proc/self/cgroup
+}
+
 case "$1" in
     exec) NO_FORK="-F"
           if [ "$2" = "--fork" ]; then
@@ -128,6 +135,53 @@ case "$1" in
           # shellcheck disable=SC2086
           exec nsenter ${NO_FORK} -a -t "${ID:-1}" "$CMD" "$@"
           ;;
+    list-app-consoles)
+          if ! is_in_debug_service; then
+              # List can be executed only in debug service container
+              # due to missing convenient GNU tools
+              eve enter debug "eve $1"
+              exit
+          fi
+          printf "PID\tAPP-UUID\t\t\t\tCONS-TYPE\tCONS-ID\n"
+          # shellcheck disable=SC3045
+          printf "---\t--------\t\t\t\t---------\t---------\n"
+          for pid in $(pgrep qemu-system); do
+              # shellcheck disable=SC2009
+              name=$(ps -p "$pid" --no-headers -o command | grep -o '\-name\s\+[^ ]\+' | awk '{print $2}')
+              # shellcheck disable=SC2009
+              uuid=$(ps -p "$pid" --no-headers -o command | grep -o '\-uuid\s\+[^ ]\+' | awk '{print $2}')
+              dir="$KVM_RUN_DIR/$name"
+              if [ -e "$dir/prime-cons" ]; then
+                  printf "%s\t%s\tCONTAINER\t%s/cons\n" "$pid" "$uuid" "$name"
+                  printf "%s\t%s\tVM\t\t%s/prime-cons\n" "$pid" "$uuid" "$name"
+              else
+                  printf "%s\t%s\tVM\t\t%s/cons\n" "$pid" "$uuid" "$name"
+              fi
+          done
+          ;;
+    attach-app-console) [ -z "$2" ] && help
+          if ! is_in_debug_service; then
+              # Attach can be executed only in debug service container
+              # due to missing convenient GNU tools
+              eve enter debug "eve $1 $2"
+              exit
+          fi
+          CONS="$KVM_RUN_DIR/$2"
+          if [ ! -e "$CONS" ]; then
+              echo "Error: console '$2' does not exist."
+              echo "Try to use 'eve list-app-consoles'."
+              exit 1
+          fi
+          PTY=$(dirname "$CONS")/pty-$$
+          # Start socat in background to pump bytes between console socket and PTY
+          trap 'kill $SOCAT_PID 2>/dev/null' EXIT HUP INT QUIT TERM
+          socat "unix-connect:$CONS" "pty,link=$PTY" &
+          SOCAT_PID=$!
+          # Give some time to socat to create PTY
+          sleep 1
+          # Start tio
+          tio "$PTY"
+          ;;
    enter) # shellcheck disable=SC2086
           ${CTR_CMD} t exec -t --exec-id $(basename $(mktemp)) ${2:-pillar} ${3:-sh -l}
           ;;

pkg/pillar/hypervisor/kvm.go

@@ -158,6 +158,22 @@ const qemuConfTemplate = `# This file is automatically generated by domainmgr
   chardev = "charserial0"
   name = "org.lfedge.eve.console.0"
 
+{{if .DomainConfig.IsOCIContainer}}
+[chardev "charserial1"]
+  backend = "socket"
+  mux = "on"
+  path = "` + kvmStateDir + `{{.DomainConfig.DisplayName}}/prime-cons"
+  server = "on"
+  wait = "off"
+  logfile = "/dev/fd/1"
+  logappend = "on"
+
+[device]
+  driver = "virtconsole"
+  chardev = "charserial1"
+  name = "org.lfedge.eve.console.prime"
+{{end}}
+
 {{if .DomainConfig.EnableVnc}}
 [vnc "default"]
   vnc = "0.0.0.0:{{if .DomainConfig.VncDisplay}}{{.DomainConfig.VncDisplay}}{{else}}0{{end}}"

pkg/pillar/hypervisor/kvm_test.go

@@ -181,6 +181,8 @@ func TestCreateDomConfigOnlyCom1(t *testing.T) {
   name = "org.lfedge.eve.console.0"
 
 
+
+
 #[device "video0"]
 #  driver = "qxl-vga"
 #  ram_size = "67108864"
@@ -443,6 +445,8 @@ func TestCreateDomConfigOnlyCom1(t *testing.T) {
   name = "org.lfedge.eve.console.0"
 
 
+
+
 #[device "video0"]
 #  driver = "qxl-vga"
 #  ram_size = "67108864"
@@ -681,6 +685,8 @@ func TestCreateDomConfigOnlyCom1(t *testing.T) {
   name = "org.lfedge.eve.console.0"
 
 
+
+
 #[device "video0"]
 #  driver = "qxl-vga"
 #  ram_size = "67108864"
@@ -1117,6 +1123,8 @@ func domConfigArm64() string {
   name = "org.lfedge.eve.console.0"
 
 
+
+
 #[device "video0"]
 #  driver = "qxl-vga"
 #  ram_size = "67108864"
@@ -1396,6 +1404,8 @@ func domConfigAmd64FML() string {
   name = "org.lfedge.eve.console.0"
 
 
+
+
 #[device "video0"]
 #  driver = "qxl-vga"
 #  ram_size = "67108864"
@@ -1686,6 +1696,8 @@ func domConfigAmd64Legacy() string {
   name = "org.lfedge.eve.console.0"
 
 
+
+
 #[device "video0"]
 #  driver = "qxl-vga"
 #  ram_size = "67108864"
@@ -1967,6 +1979,8 @@ func domConfigAmd64() string {
   name = "org.lfedge.eve.console.0"
 
 
+
+
 #[device "video0"]
 #  driver = "qxl-vga"
 #  ram_size = "67108864"

pkg/xen-tools/Dockerfile

@@ -3,7 +3,7 @@
 FROM lfedge/eve-uefi:d821658883d6748d8bbf0d6640c62288e3ce8c6f as uefi-build
 
 FROM lfedge/eve-alpine:0f4e313d0d84ac313ea35e966def9ef96f61aafb as runx-build
-ENV BUILD_PKGS mkinitfs gcc musl-dev e2fsprogs chrony
+ENV BUILD_PKGS mkinitfs gcc musl-dev e2fsprogs chrony agetty
 RUN eve-alpine-deploy.sh
 
 RUN rm -f /sbin/poweroff /etc/mkinitfs/features.d/base.files
@@ -12,6 +12,7 @@ COPY initrd/init-initrd initrd/mount_disk.sh initrd/udhcpc_script.sh /
 COPY initrd/poweroff /sbin/poweroff
 COPY initrd/chroot2.c initrd/hacf.c /tmp/
 COPY initrd/00000080 /etc/acpi/PWRF/
+COPY initrd/eve-enter-container /bin/
 RUN gcc -s -o /chroot2 /tmp/chroot2.c
 RUN gcc -s -o /hacf /tmp/hacf.c
 RUN mkinitfs -n -F base -i /init-initrd -o /runx-initrd

pkg/xen-tools/initrd/base.files

@@ -1,4 +1,6 @@
 /bin/busybox
+/bin/eve-enter-container
+/sbin/agetty
 /sbin/mke2fs
 /lib/libext2fs.so.2*
 /lib/libcom_err.so.2*