Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Cool Stuff for ELKS #1366

Closed
ghaerr opened this issue Jul 12, 2022 · 88 comments
Closed

New Cool Stuff for ELKS #1366

ghaerr opened this issue Jul 12, 2022 · 88 comments

Comments

@ghaerr
Copy link
Owner

ghaerr commented Jul 12, 2022

I'm opening this issue to talk about various projects I'm working on, most of which are running, but not fully integrated into the ELKS build yet. I'm interested to know how interested people might be in each. Here's the first three projects:

  • ELKS remote console display
  • D-Flat Text UI windowing toolkit
  • Debug stack traces for ELKS executables

I'll write up each of these separately.

@ghaerr
Copy link
Owner Author

ghaerr commented Jul 12, 2022

ELKS Remote Console

This little program allows one to login through a serial or telnet connection to ELKS (using a macOS or Linux terminal program, for instance), and display what is currently displayed on the ELKS console. This could be useful to see remote kernel messages, or just see exactly what is displayed in full color, using the ROM BIOS 437 character set.

For example, here's a login from macOS Terminal over emulated serial on QEMU, just after boot:
Screen Shot 2022-07-11 at 9 08 23 PM

The display is updated every second, and the program is terminated with ^C.

The cons program reads the PC text RAM contents directly and converts the characters into UTF-8 encoded Unicode values, along with interpreting the PC character attributes (color, reverse video, etc) and translating those into ANSI escape sequences.

The current implementation just displays the console screen, it doesn't try to insert keystrokes, etc.

@ghaerr
Copy link
Owner Author

ghaerr commented Jul 12, 2022

D-Flat Text User Interface Library

D-Flat is an old (1995 era) text user interface that allowed overlapping window-based text user interfaces to be written on DOS. I've ported it to UNIX/Linux, and now to ELKS. It is very well written, but unfortunately kind of large, and requires medium model (large code) for compilation. The underlying API is message-based, and very similar to Win32, with SendMessage and window procedures.

Here's a screenshot of the D-Flat memopad program, which allows for editing files:
Screen Shot 2022-07-11 at 9 12 35 PM

I'm primarily working on it as a possible TUI for the Cosmopolitan project, but thought it would be interesting to get it running on ELKS.

Now to show off the cons program more, here's the cons program running on serial macOS Terminal, while memopad happens to be running on the console:
Screen Shot 2022-07-11 at 9 13 21 PM

@ghaerr
Copy link
Owner Author

ghaerr commented Jul 12, 2022

Debug library for ELKS applications

While trying to get larger ELKS programs working, I've found it quite hard to understand what is going on when the program crashes, especially with no debugger or stack tracer. I've developed a set of routines that will allow for a running ELKS program (currently applications only, but could be fitted into the kernel itself for device drivers etc) to produce a stack backtrace, along with correct lookup of the actual C symbol names.

I've also implemented Cosmopolitan-style C function logging capability, which allows for display of every C function executed, to make it possible to visualize function call flow throughout program execution. I plan on adding nanosecond-precision timing display (on 386+ CPUs only, using the RDTSC instruction) to show function latencies as well. The idea would be to add an automatic "--ftrace" program argument that would turn on this display when an application is compiled for debug.

I took ELKS basic and added the stack trace capability on every function call, here are the first few functions called, along with their arguments. The display also shows the current and max stack used by the program:

ELKS 0.6.0

login: root
# basic
AX=0000 BX=7460 CX=0001 DX=7464 SI=7460 DI=3448 BP=7446 SP=7422
CS:IP=5540:36f0 DS=6200 ES=6200 SS:SP=6200:7422

Level Addr    BP   DI   SI   Ret  Arg  Arg2 Arg3 Arg4
~~~~~ ~~~~    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   0: 7418 => 7446 3448 7460 36f7 dead 0000 0000 0000 0000 0000 0000 0000 0000 0000 print_stack (364e,83)
   1: 7446 => 744e 2c60 7460 3448 745c 2c60 2cb0 0019 0001 7460 7464 0000 0001 7472 __cyg_profile_func_enter+35 (36f7,83)
   2: 744e => 745c 2c60 2cb0 0019 0001 7460 7464 0000 0001 7472 0000 7478 7482 748d main+b (3448,83)
   3: 745c => 0000 0001 7472 0000 7478 7482 748d 7497 74a1 74b2 0000 6162 6973 0063 _start+19 (0019,81)
   4: 0000 => 0000 0000 5825 0900 2800 2900 2b00 2d00 2a00 2f00 3d00 3e00 3c00 3c00 _start+1 (0001,81)
>main, from _start+19, stack 0/0
AX=0019 BX=0000 CX=5401 DX=0000 SI=7460 DI=2ce5 BP=7414 SP=73f0
CS:IP=5540:36f0 DS=6200 ES=6200 SS:SP=6200:73f0

Level Addr    BP   DI   SI   Ret  Arg  Arg2 Arg3 Arg4
~~~~~ ~~~~    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   0: 73e6 => 7414 2ce5 7460 36f7 dead 00a1 0019 73fe 73fe 354e 7408 1742 1c9c 5014 print_stack (364e,83)
   1: 7414 => 7442 0000 7460 2ce5 0000 3448 0000 0000 0000 735f 6174 7472 312b 0000 __cyg_profile_func_enter+35 (36f7,83)
   2: 7442 => 744e 0000 7460 345b 0000 46b8 745c 2c60 2cb0 0019 0001 7460 7464 0000 tty_isig+b (2ce5,83)
   3: 744e => 745c 2c60 2cb0 0019 0001 7460 7464 0000 0001 7472 0000 7478 7482 748d main+1e (345b,83)
   4: 745c => 0000 0001 7472 0000 7478 7482 748d 7497 74a1 74b2 0000 6162 6973 0063 _start+19 (0019,81)
   5: 0000 => 0000 0000 5825 0900 2800 2900 2b00 2d00 2a00 2f00 3d00 3e00 3c00 3c00 _start+1 (0001,81)
|>tty_isig, from main+1e, stack 25/25

I am also thinking of a use where a function backtrace would be displayed whenever ^C (SIGINT) is typed, to show what the program is doing at the time of interrupt. This would be very useful for debugging hung programs.

The instrumentation facilities used to implement function call tracing require a patch to the ELKS compiler toolchain, and the stack trace capabilities require some special link handling which adds debug symbol information into the current executable, both of which are rather complicated at the moment.

In addition, I have some library code that can interpret memory bytes and display as 8086 disassembly, running directly on ELKS. This could be used within an application, or added to the hd or a new disasm or monitor/debug program, for better visibility of very low-level stuff while running on ELKS. The hd program is going to be enhanced to display codepage 437 (ROM BIOS) characters on a terminal display, to better see binary values as part of a hex dump.

@Mellvik
Copy link
Contributor

Mellvik commented Jul 12, 2022 via email

@tyama501
Copy link
Contributor

Hello @ghaerr ,

Thank you for very cool stuffs!
I started to port nano-X to pc-98 but it seems harder than I thought so it would be nice to have the debug library.

D-Flat is also interesting to see applications like DOS edit on ELKS but I'm not sure it will work for PC-98 too.

@ghaerr
Copy link
Owner Author

ghaerr commented Jul 12, 2022

Hello @tyama501,

I started to port nano-X to pc-98 but it seems harder than I thought

It may be easiest to just copy nano-X/drivers/scr_bios.c to scr_pc98.c, and plan on having all graphics draw code in that single file. Rename all the functions from VGA_ to pc98_, and then start with just writing the pc98_open() and pc98_close() routines which should switch the screen to and from graphics mode, using the same code from BASIC.

You should then be able to write the very basic pc98_drawpixel, pc98_drawhline, pc98_drawvline and pc98_fillrect functions, which do not depend on an internal graphics format. (HAVEBLIT should be set to 0).

After that, in order to support BLIT, we would need to make a decision as to how nano-X should store the graphics data internally, which is set with the psd->app and other statements in pc98_open().

If you'd like to share what you have, I'm happy to help you get the screen driver working. The mouse driver, but default uses the serial port, which isn't yet implemented for PC-98, so you might want to compile in drivers/mou_null.c.

D-Flat is also interesting to see applications like DOS edit on ELKS but I'm not sure it will work for PC-98 too.

What is the character set for byte values 128 - 255 in PC-98 character ROM? These won't be codepage 437, so that could require low-level changes to D-Flat for running on console. However, I am thinking having the library code automatically switch to using ANSI ESC sequences when running on telnet or serial, and having the CP 437 codes converted to unicode for that case (the same was cons works, basically).

Thank you!

@ghaerr
Copy link
Owner Author

ghaerr commented Jul 12, 2022

Hello @Mellvik,

Apropos nanosecond resolution - don't you see a challenge in the way the output would affect the timing?

Yes, the trace and timer code will definitely add overhead to all function tracing, but the idea is it will be a fixed amount, and will adjust all timings by that same fixed amount, so bottleneck latencies can still be easily spotted. The routine could calculate its own overhead and subtract that from the displayed time for better accuracy (I've added that thought to my list).

Disasm: Sounds very useful, but I didn't understand how it may be used.

I think the most useful case would be some kind of debug/monitor that would trap to a prompt (interrupting the running program from a SIGINT), then producing a backtrace and allow the user to dump register or specify a seg:off memory address to start disassembly from. As a library routine, it would also be useful to write programs to disassemble various files at times. Another use could be instruction tracing, by allowing a single instruction to execute on real hardware, followed by an INT 3 back to the monitor. This would work because the disassembler has the ability to compute how long a given instruction sequence is.

I'm not quite sure where to put all this - I suppose just adding the the C library would be OK, except these are all non-standard routines. So perhaps a separate library... not sure yet.

I'm also working on being able to compile and link programs outside of the ELKS source tree - this is too hard currently, as there's no easy way to get all the compiler options and C library. Once this is done it will be lots easier to contribute programs to ELKS, without having to become intimately familiar with the kernel and application source trees.

Thank you!

@Mellvik
Copy link
Contributor

Mellvik commented Jul 12, 2022 via email

@tyama501
Copy link
Contributor

Hello @ghaerr

It may be easiest to just copy nano-X/drivers/scr_bios.c to scr_pc98.c, and plan on having all graphics draw code in that single file.

I have copied vgaplan4.c to vgaplan4_pc98.c and I have been modifiying ega_drawpixel to draw directly to 0xA800, 0xB000, 0xB800, 0xE000 VRAM after changing to the graphic mode from basic but your idea sounds easier for the first step.

the serial port, which isn't yet implemented for PC-98, so you might want to compile in drivers/mou_null.c.

I have noticed this, and I commented out GdOpenMouse in srvmain.c for now.
OK, I will use mou_null instead.

For PC-98 there is no serial mouse, and the mouse is attached to 8255 I/O port.
So, maybe we need to write the driver with polling or interrupt after the screen driver worked.

D-Flat is also interesting to see applications like DOS edit on ELKS but I'm not sure it will work for PC-98 too.

What is the character set for byte values 128 - 255 in PC-98 character ROM?

I poked to 0 - 255 directly to text vram with basic :)
It seems there are single byte katakana-characters and some others.

pokeb_0to255_text

Thank you.

@ghaerr
Copy link
Owner Author

ghaerr commented Jul 13, 2022

Hello @tyama501,

I have copied vgaplan4.c to vgaplan4_pc98.c and I have been modifiying ega_drawpixel to draw directly to 0xA800, 0xB000, 0xB800, 0xE000 VRAM after changing to the graphic mode

I see. I suppose it depends on how close the PC-98 graphics hardware is to EGA - EGA is terrible, and requires multiple mask and plane register to be set for each pixel. If that's the case for PC-98, perhaps staying with your present approach is better. Another thought would be to only support mono (2-color) mode initially, is that possible, then possibly not having to write to each memory plane for each pixel.

After getting running, you will want to "define NDEBUG" to remove the assert() code, as it slows things down considerably.

For PC-98 there is no serial mouse, and the mouse is attached to 8255 I/O port.
So, maybe we need to write the driver with polling or interrupt after the screen driver worked.

When you use mou_null, this should happen automatically. Afterwards, a PC-98 mouse driver can be written.

In general, nano-X busy-loops by polling, so this shouldn't be a problem unless the polling itself is quite slow.

It seems there are single byte katakana-characters and some others.

I can see it's definitely not codepage 437, but does include the box line draw characters. There doesn't seem to be the 50% gray shaded character that is used for the background of D-Flat windows, but that isn't a big problem, a space can be used.

Should we get more interested in actually running D-Flat on PC-98, we can develop a translation table for the different code page, both for D-Flat use, as well as for the cons unicode translation.

Thank you!

@toncho11
Copy link
Contributor

toncho11 commented Jul 14, 2022

Hi @ghaerr,

I am for D-Flat project - it seems it can produce a nice DOS editor and some other applications as well. We can have a developer section one day: "Programming for ELSK" using:

  • pure C code
  • using X-nano
  • using D-Flat TUI
  • BASIC

@ghaerr
Copy link
Owner Author

ghaerr commented Jul 18, 2022

D-Flat and its text-mode user interface library demo is now running on ELKS connecting from serial or telnet terminal sessions. Here's a video showing it in operation. In this particular case, all I/O is based on ANSI terminal escape sequences, so the TUI display is available to many remote systems, using any modern terminal emulator, such as macOS Terminal or Linux xterm-256.

It also runs on the console, but this requires more setup as a mouse driver that spits out ANSI sequences needs to be installed, unless the program can be entirely operated from the keyboard.

The video is too big for Github upload, here's a link to my dropbox for viewing.

@toncho11
Copy link
Contributor

Nice job @ghaerr !!!
So it can run natively (without any remote sessions) on ELKS, but one needs to use the keyboard shortcuts.

D-Flat and its text-mode user interface library demo is now running on ELKS connecting from serial or telnet terminal sessions. Here's a video showing it in operation. In this particular case, all I/O is based on ANSI terminal escape sequences, so the TUI display is available to many remote systems, using any modern terminal emulator, such as macOS Terminal or Linux xterm-256.

It also runs on the console, but this requires more setup as a mouse driver that spits out ANSI sequences needs to be installed, unless the program can be entirely operated from the keyboard.

The video is too big for Github upload, here's a link to my dropbox for viewing.

@Mellvik
Copy link
Contributor

Mellvik commented Jul 18, 2022 via email

@ghaerr
Copy link
Owner Author

ghaerr commented Jul 18, 2022

Can the interface be operated w/o a pointing device?

Yes, I'm sure it could, but not certain as to what has been coded in D-Flat. D-Flat is a somewhat large text user interface (TUI) library, by ELKS' standards. The memopad demo, which includes all the TUI controls (scrollbars, listboxes, popups, comboboxes, edit boxes, etc) is 110k - huge. D-Flat is IBM SAA/CUA compliant as of their 1987 standard.

However, the real purpose of the work is mostly R&D, and under the hood portions of the following are already implemented, or being considered:

  • Working medium model ELKS applications (a toolchain bug was recently fixed getting memopad to load properly).
  • Writing TUI applications that compile and work almost identically between macOS, Linux and ELKS.
  • Unicode support, which allow ELKS to run applications that can display unicode (using UTF-8) on serial or telnet connections. In particular, the ability to display BIOS character ROM codepage 437 characters external to the console is a big win.
  • Possible solutions to long-standing character display problems such as German Umlauts are not correctly shown #1175 because the PC ROM character set isn't the same across machines internationally.
  • Enhancement of incoming and outgoing ANSI escape sequence support for ELKS console and terminal sessions, including accepting a wider ranger sequences on direct console for UNIX application compatibility, as well as ANSI mouse and window sizing support being put in library routines.
  • A possible ability for an application to know whether its running on direct console or serial/telnet terminal, and to automatically perform character conversions via Unicode and/or ANSI sequences for visual compatibility. (For instance, the cons console display program, and planned hd display enhancements to see the full CP437 character set).
  • Possible support for arbitrary unicode on non-console applications. This is important, as the rest of the world is now running on unicode, and ANSI sequences allow unicode display on all modern terminal emulators.
  • Development of multiple layers of TUI support, from automatic mouse/keyboard support without termcap or ncurses (using ANSI/xterm standard) for building portable TUI applications, the ability to write characters/attributes to the screen directly for speed, all the way to heavier-weight libraries like D-Flat.
  • Test programs that allow for easier unit-testing of various library components.
  • Testing of the kernel and C library implementation by using larger pieces of software (more incompatibilities have been found).

All this said, the fact that ELKS is basically running almost all of D-Flat identically to its UNIX counterpart is a testament to how far ELKS has come in the last few years - its pretty amazing our 16-bit system can keep up and run more modern UNIX 32- and 64-bit programs, thanks to everyone's work :)

@toncho11
Copy link
Contributor

toncho11 commented Jul 19, 2022

@ghaerr Are you planning on submitting D-Flat as part of the bin folder? Maybe only for the HDD images?

@toncho11
Copy link
Contributor

toncho11 commented Jul 19, 2022

@ghaerr Actually as a binary release it will be only memopad?

@ghaerr
Copy link
Owner Author

ghaerr commented Jul 19, 2022

Hello @toncho11,

My plan would be to consider adding the D-Flat source code as a text user interface library, along with a number of other helpful library routines, somewhere in the ELKS source tree. The library would be automatically built so that others could build their own TUI applications. The D-Flat library includes only a single demo memopad that displays all of its capabilities which would also be copied to the /bin directory for any images 2.88M and larger.

The real problem with most TUI and GUI libraries is that it isn't trivial to build an application using them. Frankly, it isn't real clear to me what anyone might build with a D-Flat TUI. (Given that the kilo and nano editors don't work well, we renamed the MINIX screen mined to edit, and it seems to work quite well, so we don't need a screen editor in D-Flat). Even though D-Flat is well thought-out and well-written, it is large by ELKS standards, and programs produced using the library won't fit well on floppy disks with the rest of the standard distribution.

Thus, D-Flat itself is mostly a research project, which (described above) is helping to illuminate some things that might want to be considered next for development. That said, it's opening the doors for more discussion on what we should next add to ELKS. Thank you for your comments!

@Mellvik
Copy link
Contributor

Mellvik commented Jul 20, 2022 via email

@ATroubleshooter
Copy link

Thus, D-Flat itself is mostly a research project, which (described above) is helping to illuminate some things that might want to be considered next for development. That said, it's opening the doors for more discussion on what we should next add to ELKS. Thank you for your comments!

Considering D-Flat, in regards to as what should be added next - probably, a simplistic lightweight twin-panel file manager, based on D-Flat?

@ghaerr
Copy link
Owner Author

ghaerr commented Jul 21, 2022

Considering D-Flat, in regards to as what should be added next - probably, a simplistic lightweight twin-panel file manager, based on D-Flat?

That's a nice idea, but potentially a lot of work to (re)write a nice file manager. What would help would be something like a 16-bit version of Midnight Commander, which was discussed a bit in #1117, but apparently not available. With the D-Flat lower level ANSI/xterm input/output, we could eliminate the need for the huge and clunky ncurses library, and only use xterm/ANSI sequences, which would display properly on ELKS and most all Linux/macOS systems. That is, an older ncurses-based file manager could be slimmed down and possibly more easily ported by replacing ncurses with the D-Flat lower level.

Does anyone have any ideas about an older, small file manager that might be portable?

@ghaerr
Copy link
Owner Author

ghaerr commented Jul 21, 2022

Looking around for other "small" text-based file managers, I finally found two, both based on ncurses: nnn (https://github.com/jarun/nnn) which is highly supported and does lots of stuff. It purports to running well on embedded systems. Here's a screenshot:
nnn screenshot

However, in reading a comparison article, nnn apparently uses far less memory than other Linux-based file managers, "only" 3.5MB of RAM!!! (compared to 9.8MB for Midnight Commander, and 51MB for the largest):
nnn-Memory The nnn executable on macOS is 131K with no options enabled.

I guess RAM is cheap.

Looking further, I find noice (https://github.com/RichardHyde/noice), which it turns out nnn is forked from. It compiled quickly on macOS, and is a 29K executable. Its development stopped in 2015, but it looks very similar, just doesn't use any color display:
noice screenshot

The nnn source is 8800 lines, and noice is 1000. Looking at noice, it seems it could be ported to ELKS quickly, with an initial version using just the arrow keys (and/or mouse on a remote terminal), using the low-level D-Flat framework I just developed. Later, a version could be possibly inserted into a D-Flat frame to look cooler. I haven't looked at the memory usage yet for noice. I may take a pass at porting noice to ELKS if people are interested.

@ghaerr
Copy link
Owner Author

ghaerr commented Jul 22, 2022

I looked further into noice and found the original project (https://git.2f30.org/noice), which was last updated in 2019. It is a quite small and nifty, portable file manager.

I ported this to ELKS as our new fm file manager, and wrote an ncurses library emulation layer for it, which allowed fm to run directly on the console or from a serial/telnet terminal. The file manager is only 13K bytes, so it should run well on floppies.

Here's a screen recording of it in operation. I plan on porting the lower layer to be able to use the mouse or function keys on the console or serial/telnet terminal, and then ultimately should be pretty straightforward to put into a D-Flat window. Pretty cool!

ELKS.file.manager.mov

Thanks for the idea @ATroubleshooter :)

@Mellvik
Copy link
Contributor

Mellvik commented Jul 22, 2022 via email

@ATroubleshooter
Copy link

Great job @ghaerr, keep it up!

@ghaerr
Copy link
Owner Author

ghaerr commented Jul 26, 2022

Update on new File Manager for ELKS and UNIX:

Significant progress has been made, the the project now compiles and runs identically on ELKS console, ELKS terminals, as well as UNIX (tested currently only on macOS Terminal). It seems to work pretty well, and for now have created its own project, before integrating directly into ELKS.

Here's a screenshot running on macOS (which pretty much looks identical to the ELKS terminal version):
FM terminal

The file manager has lots of features for its small size, but here are the basic keystrokes:

RightArrow/Enter - Enter directory or display file
LeftArrow/Backspace - Back up one directory
Up/DownArrow or mouse scroll wheel - move cursor
a-z - goto file matching letter pressed
Z or ^C - quit
D - directories displayed first
T - sort by time
S - sort by size
C - change directory
E - edit file
. - display hidden files
~ - home directory
? - help

The program automatically decompresses .Z files for display, and will run man on man page files in compressed or normal format. more is used to display files.

After cloning the source tree, type make for the Linux version, or make -f Makefile.elks for the ELKS version, then type ./fm.

I've worked hard to make the program as small as possible, which is pretty small since ncurses and regex libraries were removed. fm requires and ANSI (or xterm-256) terminal to run.

The ELKS version is 20k and UNIX 40k. Let me know what you think. At some later point I may wrap the output in D-Flat to look more like Midnight Commander.

Enjoy!

@toncho11
Copy link
Contributor

Great job @ghaerr !

@toncho11
Copy link
Contributor

I see edit is possible.
Here are suggestions for future versions:

  • configure editor (which is the default now?)
  • copy files with progress bar
  • delete and move both files and directories

Looking back I like Norton Commander, but PC Tools is the one that impressed me the most back then. PCtools is not double screen. A new window pops up and you can select directory tree or drive (I think).

@ghaerr
Copy link
Owner Author

ghaerr commented Jul 26, 2022

configure editor (which is the default now?)

vi is the default, but any editor can be specified using the EDITOR= environment variable.

copy files with progress bar
delete and move both files and directories

Good ideas, but fm is currently just single-paned. I'll wait and see whether people actually use it before spending too much time on it. With the under-the-hood work on the keyboard and mouse input for D-flat, it was really good timing to work on another program that heavily used the mouse and arrow keys. D-flat doesn't yet support the mouse wheel, and I'm going to jump back into that for bit and try to get a version others can play with on ELKS or UNIX.

@ghaerr
Copy link
Owner Author

ghaerr commented Jul 28, 2022

Update on D-Flat for ELKS and UNIX:

A separate project has been created for D-Flat, where it can be compiled for macOS, Linux or ELKS. Things have been progressing nicely, and it now automatically sizes to the open Terminal on startup, just like the fm File Manager. Both projects share the same new "unikey" ANSI keyboard/mouse library, which also supports unicode character conversions (currently only for code page CP437). The screen updates are FAST.

Here's D-Flat's memopad running on macOS. The display looks identical when run from ELKS serial or telnet:
D-Flat_Text_User_Interface_on_UNIX

The new unikey library supports ANSI mouse sequences from macOS and Linux terminal programs emulating xterm-256, and I plan on adding the mouse sequence support to ELKS running QEMU over an emulated serial port. That would allow for operation of QEMU console programs using the mouse from your host computer or trackpad to operate it. On real hardware, the serial mouse is already supported, but not yet built into the unikey library, which would allow for a new class of programs to be developed for ELKS - text based user interfaces which could be sped up greatly just by using the mouse wheel for fast scrolling, for instance.

If you have time, take a look at the project and compile it. Because both D-Flat and File Manager are being developed for both UNIX and ELKS, I'm thinking that adding them as a Git submodule (i.e. a repo within the ELKS repo) in the future, which would allow for automatic compilation and inclusion on the ELKS disk images.

@cocus
Copy link
Contributor

cocus commented May 20, 2023

Hi all, this looks so cool! In fact, it's so cool, that I dusted off my 8018x and flashed it with today's tip of master.
The text-ui came pre-enabled by default, but that doesn't fit on my ROMFS (I think it's because the actual kernel is placed right after the ROMFS and I should move it a little bit); but I was able to copy the binaries to an SD and run them.
I was able to run the fm fine, although really slow (a little bit faster over ethernet for sure!), but cons doesn't do anything interesting, except writing colored-characters all over the screen (either via serial or telnet, with putty or picocom):
image
I would like to run the memopad or something cool, and feel the "d-flat" experience (which I was looking for like 4 months ago, never knew the name of this "windowing" scheme).
Do I have to do something special in order for this to work?
Thank you!

@ghaerr
Copy link
Owner Author

ghaerr commented May 20, 2023

Hello @cocus,

Nice to hear from you, how have you been doing?

but cons doesn't do anything interesting, except writing colored-characters all over the screen

The reason cons doesn't work on your 8018x is that it reads the IBM PC BIOS display screen area at B800:0, which appears to be either random memory or non-existent on your system. It would be possible to hack the direct console with some work, which could then run headless and then use cons to display the pseudo-console, but that's probably not worth the effort, and the keyboard would have to be dealt with separately.

I would like to run the memopad or something cool, and feel the "d-flat" experience

Because of its size, D-Flat and memopad are built in a seperate repository, for both UNIX and ELKS. You can take a look at it over here and build the ELKS version using make -f Makefile.elks.

The text-ui came pre-enabled by default

What are you referring to here?

Thank you!

@cocus
Copy link
Contributor

cocus commented May 20, 2023

Nice to hear from you, how have you been doing?

Pretty good! Thank you! How are you doing? I see you were really busy with elks!

The reason cons doesn't work on your 8018x is that it reads the IBM PC BIOS display screen area at B800:0, which appears to be either random memory or non-existent on your system. It would be possible to hack the direct console with some work, which could then run headless and then use cons to display the pseudo-console, but that's probably not worth the effort, and the keyboard would have to be dealt with separately.

Ah! I thought cons was required for memopad to run. I was mistaken. Yes, I agree, completely useless to do this on the headless targets

What are you referring to here?

Oh, if I copy the 8018x.config as my main .config and run the menuconfig, I see that the text-ui was enabled by default on the userland.

Because of its size, D-Flat and memopad are built in a seperate repository, for both UNIX and ELKS. You can take a look at it over here and build the ELKS version using make -f Makefile.elks.

Ah! cool. I just built it (I had to change the path to my elks TOPDIR under the Makefile.elks) but the binary doesn't run on my board, I always get a

# ./memopad
0@�@^: not found
./memopad: 2: Syntax error: ")" unexpected

the file seems to be okay. I copied it to the SD card because it won't fit on the rootfs romfs.

@ghaerr
Copy link
Owner Author

ghaerr commented May 20, 2023

the binary doesn't run on my board,

Oh, memopad is built in medium model, since its code size > 64k. That means the kernel needs to have the a.out medium model extension enabled: set CONFIG_EXEC_MMODEL=y and rebuild the kernel. (Sorry if that's a pain to reflash!)

I see that the text-ui was enabled by default on the userland.

I'm sorry I'm being a little dense here, but I still don't know what text-ui is referring to, I don't recall having ever seen that term used within ELKS.

@cocus
Copy link
Contributor

cocus commented May 20, 2023

Oh, memopad is built in medium model, since its code size > 64k. That means the kernel needs to have the a.out medium model extension enabled: set CONFIG_EXEC_MMODEL=y and rebuild the kernel. (Sorry if that's a pain to reflash!)

Okay, I'll test this. What are the downsides of enabling this? can I do it and keep it as the default?

I'm sorry I'm being a little dense here, but I still don't know what text-ui is referring to, I don't recall having ever seen that term used within ELKS.

It's the name set on the menuconfig for tui:
image

@ghaerr
Copy link
Owner Author

ghaerr commented May 20, 2023

What are the downsides of enabling this? can I do it and keep it as the default?

There are no downsides. Its only a config option because of the extra space taken up in the kernel, which is small. Perhaps it should be on by default. (Though not many programs are medium model at this point, I think memopad is the only one :)

It's the name set on the menuconfig for tui:

Geez, thanks. Yes, I named that and promptly forgot what I called it. Are you thinking it should be off by default for 8018x?

@cocus
Copy link
Contributor

cocus commented May 20, 2023

There are no downsides. Its only a config option because of the extra space taken up in the kernel, which is small. Perhaps it should be on by default. (Though not many programs are medium model at this point, I think memopad is the only one :)

Well, I enabled it, but it still doesn't run. Could it be because I don't have enough memory? meminfo reports 360kb free.

# ./memopad 
# 
# ./memopad 
# 

Geez, thanks. Yes, I named that and promptly forgot what I called it. Are you thinking it should be off by default for 8018x?

Well, I think it might be a good idea to turn it off by default, just for the sake of reclaiming some bytes on the romfs

@ghaerr
Copy link
Owner Author

ghaerr commented May 20, 2023 via email

@cocus
Copy link
Contributor

cocus commented May 20, 2023

Does trying to run memopad act like a shell script command failure, or does the shell return an exec error? If the former, the a.out magic number is not correct, which means either no kernel support or bad executable. Post your memopad and I’ll run it when I get home. Push text ui off on your open PR and we’ll commit it there.

I just pushed the config with the tui disabled, but I need to enable the medium model config. I'll do that in a second.

When I run it, no error is shown:

# cd /mnt
# ./memopad
# echo $?
0

@ghaerr
Copy link
Owner Author

ghaerr commented May 21, 2023 via email

@ghaerr
Copy link
Owner Author

ghaerr commented May 21, 2023

Hello @cocus,

I enabled it, but it still doesn't run. Could it be because I don't have enough memory? meminfo reports 360kb free.

It looks like memopad needs an 84k code segment and is requesting (and getting) a 65k data segment (see second screenshot for these), which matches the 147k or so shown by ps:
DF ps 1

DF heap

Can you post a meminfo of your system after boot and login? That will show what the free memory looks like.

I'm also trying to track down whether sh or sash will display an error when errno=ENOMEM is returned from exec, which should be happening, but likely is not.

(I'd like to add an option to meminfo that would make it easier to show just application used/available memory, rather than all the other stuff... its getting kind of long).

Thank you!

@cocus
Copy link
Contributor

cocus commented May 21, 2023

Can you post a meminfo of your system after boot and login? That will show what the free memory looks like.

# meminfo
  HEAP   TYPE  SIZE    SEG   TYPE    SIZE  CNT
  ab7e   SEG     16   1080   CSEG    7056    1
  ab9a   INT      6
  abac   INT      6
  abbe   INT      6
  abd0   INT      6
  abe2   BUFH   216
  acc6   TTY     80
  ad22   TTY     80
  ad7e   SEG     16   1239   DSEG    5808    1
  ad9a   SEG     16   13a4   free   24864    0
  adb6   SEG     16   25e6   DSEG   24864    1
  add2   SEG     16   2bf8   DSEG    2352    1
  adee   SEG     16   19b6   CSEG    3632    1
  ae0a   SEG     16   1a99   free    2320    0
  ae26   SEG     16   1b2a   CSEG   43968    1
  ae42   SEG     16   2c8b   free  341840    0
  ae5e   free 20898
  Heap/free   21646/20898 Total mem  456704
  Memory usage  446KB total,   86KB used,  360KB free
# /mnt/memopad
# 

@ghaerr
Copy link
Owner Author

ghaerr commented May 21, 2023

Thanks @cocus. You're running this from the external headless console to Putty, right? Can you do a ps and tell me what the TTY number is you're running on ? memopad does try to do something slightly different, by not translating all the CP 437 characters to ANSI Unicode sequences, when running on /dev/tty1 but that shouldn't affecting this. I can't yet determine whether its running, crashing, or the kernel is rejecting it with ENOMEM and /bin/sh isn't saying so. The meminfo above shows plenty of room for the executable. If you'd like to post memopad here, I'll try running it on QEMU (or you can try that also - the D-Flat Makefile.elks copies it to ELKS /root, I happen to notice).

@cocus
Copy link
Contributor

cocus commented May 21, 2023

Thanks @cocus. You're running this from the external headless console to Putty, right? Can you do a ps and tell me what the TTY number is you're running on ?
Yes, this is through the headless console, The same happens when I run it through telnet, although with less available RAM.
I'm using picocom, but the same happens with Putty.

# ps
  PID   GRP  TTY USER STAT CPU  HEAP  FREE   SIZE COMMAND
    1     0      root    S   0  3072  2016  12864 /bin/init 
    6     6    1 root    S   1  1150 13234  68832 -/bin/sh 
   11     6    1 root    R   0  1024  1176  11264 ps

memopad does try to do something slightly different, by not translating all the CP 437 characters to ANSI Unicode sequences, when running on /dev/tty1 but that shouldn't affecting this. I can't yet determine whether its running, crashing, or the kernel is rejecting it with ENOMEM and /bin/sh isn't saying so. The meminfo above shows plenty of room for the executable. If you'd like to post memopad here, I'll try running it on QEMU (or you can try that also - the D-Flat Makefile.elks copies it to ELKS /root, I happen to notice).

Here it is:
https://www.dropbox.com/s/fhb81xqzvkhk2oe/memopad?dl=0

@ghaerr
Copy link
Owner Author

ghaerr commented May 21, 2023

Thanks, your version runs fine on QEMU, and is the same as mine.

I see that your system is running headless on /dev/tty1. That means memopad (the D-Flat library) is thinking the system is running on the ELKS console.

Try rebuilding memopad with the following patch and rerunning it:

diff --git a/tty.c b/tty.c
index a519134..030a84f 100644
--- a/tty.c
+++ b/tty.c
@@ -110,6 +110,7 @@ int tty_init(enum ttyflags f)
         once = 1;
         memcpy(&t, &oldterm, sizeof(t));
         iselksconsole = tty_iselksconsole(1);
+        iselksconsole = 0;
     }
     tty_enable_unikey();
     if (flags & FullBuffer)

@cocus
Copy link
Contributor

cocus commented May 21, 2023

I changed that, re build it, and tested it again, but this time we're back with the old errors:

# /mnt/memopad
0@�@^: not found
/mnt/memopad: 2: Syntax error: ")" unexpected

I'm not sure why, the medium model thing is enabled on my kernel.

@ghaerr
Copy link
Owner Author

ghaerr commented May 21, 2023

Well, something is goofed up for sure. To determine if you've got a valid executable you can run chmem:

ELKS 0.6.0

login: root
# chmem ./memopad 
 TEXT  FTEXT   DATA    BSS   HEAP  STACK  TOTDATA  TOTAL
42736  41520  24128   6944  65535      0    65520 149776 ./memopad
# 

If it shows as valid (with approx 84k text and heap set to 65K which means max heap), then it means that your kernel isn't running CONFIG_EXEC_MMODEL. Otherwise, we've got a build issue with make elks over at D-Flat.

@cocus
Copy link
Contributor

cocus commented May 21, 2023

If it shows as valid (with approx 84k text and heap set to 65K which means max heap), then it means that your kernel isn't running CONFIG_EXEC_MMODEL. Otherwise, we've got a build issue with make elks over at D-Flat.

i've got the same output as you, so the MMODEL is not set then? let me clean everything on the elks topdir and re-build the images

# /mnt/chmem /mnt/memopad 
 TEXT  FTEXT   DATA    BSS   HEAP  STACK  TOTDATA  TOTAL
42736  41520  24128   6944  65535      0    65520 149776 /mnt/memopad
# /mnt/memopad
0@�@^: not found
/mnt/memopad: 2: Syntax error: ")" unexpected

@ghaerr
Copy link
Owner Author

ghaerr commented May 21, 2023

Sounds like CONFIG_EXEC_MMODEL is not set. However, your chmem should be slightly different than mine with the iselksconsole = 0; patch included.

@cocus
Copy link
Contributor

cocus commented May 21, 2023

image

hey hey! this is so painfully slow over serial, that I might die! but it works!!!! how cool is this!

I see the uart TX led constantly on, even while not doing absolutely anything on the terminal... it might be expected I guess

image

@ghaerr
Copy link
Owner Author

ghaerr commented May 21, 2023

this is so painfully slow over serial, that I might die! but it works!!!! how cool is this!

Well it does look cool, even it it's hellishly slow?! Kind of neat that your ROM based 8018x is computing all this :)

What baud rate are you running? I'm not yet certain whether the bottleneck is just the 80c188eb calculating D-Flat stuff, or perhaps a dumb virtual screen to ANSI sequence conversion routine I wrote. I haven't actually ever run this over actual serial, only emulated on a fast laptop over pseudo terminals!

I see the uart TX led constantly on, even while not doing absolutely anything on the terminal...

I just happened to notice something similar when coming up with the iselksconsole patch. I'll investigate that some more and also try to come up with a mechanism other than the current patch for allowing D-Flat to run on ELKS console and serial.
It is also possible the continuous transmitting could be part of the speed problem you're seeing.

@cocus
Copy link
Contributor

cocus commented May 21, 2023

What baud rate are you running? I'm not yet certain whether the bottleneck is just the 80c188eb calculating D-Flat stuff, or perhaps a dumb virtual screen to ANSI sequence conversion routine I wrote. I haven't actually ever run this over actual serial, only emulated on a fast laptop over pseudo terminals!

I'd say it's both the CPU and the serial. It's running at 9600 by default (since the headless driver configures it by default, even if the actual driver supports faster baudrates). I've tried 38400 bps before, and it works, but the driver always initializes it at 9600 :(.

Running it thru telnet is "somewhat" faster, but still slow, so you can get an idea that the problem is the CPU more or less. Even if it runs at 16MHz.

I just happened to notice something similar when coming up with the iselksconsole patch. I'll investigate that some more and also try to come up with a mechanism other than the current patch for allowing D-Flat to run on ELKS console and serial.
It is also possible the continuous transmitting could be part of the speed problem you're seeing.

Yeah, no problems, just seeing this is amazing. Thank you for your efforts!

@ghaerr
Copy link
Owner Author

ghaerr commented May 22, 2023

I've tried 38400 bps before, and it works, but the driver always initializes it at 9600 :(.

You're welcome to submit a patch for CONFIG_ARCH_8018X to default serial baud rate to 38400. I'd do it for IBM PC except I think others may guess/expect 9600.

I'd be interested to know more accurately how much the baud rate affects the results here. Perhaps even running 57600 would help. If the speedup is significant, its not CPU bound. Another idea would be to implement interrupt-driven transmit, instead of busying looping for output. And memopad shouldn't be looping or continuously outputting byte streams, I'll look into that on QEMU.

Now you need a mouse (somehow). Is that possible on that system (there's only one serial port, right)?

@cocus
Copy link
Contributor

cocus commented May 22, 2023

You're welcome to submit a patch for CONFIG_ARCH_8018X to default serial baud rate to 38400. I'd do it for IBM PC except I think others may guess/expect 9600.

I'd be interested to know more accurately how much the baud rate affects the results here. Perhaps even running 57600 would help. If the speedup is significant, its not CPU bound. Another idea would be to implement interrupt-driven transmit, instead of busying looping for output. And memopad shouldn't be looping or continuously outputting byte streams, I'll look into that on QEMU.

It'd be nice to know how to force the default speed to 38400, haven't figured how to do that myself.

Now you need a mouse (somehow). Is that possible on that system (there's only one serial port, right)?

It has two serial ports, but the secondary one was never enabled, nor any drivers added to it. However, using the main serial port and a console emulator, I can use the mouse without any issues! It works fine! I was even able to drag and drop a window, slowly as hell, but they moved!

@ghaerr
Copy link
Owner Author

ghaerr commented May 22, 2023

It'd be nice to know how to force the default speed to 38400, haven't figured how to do that myself.

The 8018x port is using a special console implemented in elks/arch/i86/drivers/char/console-serial-8018x.c. In that file, the console_init function calls update_port which references the associatedstruct tty found in ntty.c. In ntty.c the struct termios def_vals is initialized with B9600 | CS8 which could be ifdefed CONFIG_ARCH_8018X and set to B38400 | CS8 (or perhaps set a DEFAULT_BAUD earlier in the file with an ifdef there).

[EDIT: Try 57600 or more, it would be interesting to see what the highest value the 8018x might support. At some point too high a baud rate may case loss of data from the mouse stream... ].

using the main serial port and a console emulator, I can use the mouse without any issues!

You mean your terminal emulator allows using a mouse, which is interpreted and then sent out to the serial input stream by D-Flat and moves the mouse? That's pretty cool, since I initially designed D-Flat to take mouse movements from the macOS Terminal, but didn't realize this could be used with a serial-based terminal emulator that interfaced with a mouse and does the same thing! Very nice!

@cocus
Copy link
Contributor

cocus commented May 22, 2023

The 8018x port is using a special console implemented in elks/arch/i86/drivers/char/console-serial-8018x.c. In that file, the console_init function calls update_port which references the associatedstruct tty found in ntty.c. In ntty.c the struct termios def_vals is initialized with B9600 | CS8 which could be ifdefed CONFIG_ARCH_8018X and set to B38400 | CS8 (or perhaps set a DEFAULT_BAUD earlier in the file with an ifdef there).

Aha, I'll try this! Good idea.

[EDIT: Try 57600 or more, it would be interesting to see what the highest value the 8018x might support. At some point too high a baud rate may case loss of data from the mouse stream... ].

I'd love to use even more than that, but the problem is the FCPU that I have (which is set by the external crystal). The baudrate generator is really off when going above 38400. Usually my FTDI USB-serial doesn't even read anything, and I've seen this on the logic analyzer that the timings are really off, as the theory said.

It's funny because some other user with a different clock could easily use the higher baudrates without issues!

You mean your terminal emulator allows using a mouse, which is interpreted and then sent out to the serial input stream by D-Flat and moves the mouse? That's pretty cool, since I initially designed D-Flat to take mouse movements from the macOS Terminal, but didn't realize this could be used with a serial-based terminal emulator that interfaced with a mouse and does the same thing! Very nice!

Yep, I did exactly that. It worked on my telnet and picocom, didn't test PuTTY but it usually works as well.

@tyama501
Copy link
Contributor

(I'd like to add an option to meminfo that would make it easier to show just application used/available memory, rather than all the other stuff... its getting kind of long).

I think so too.
It would be nice if meminfo can show names of applications like FreeDOS mem.
Is it hard to do that?

FreeDOS mem
mem_freedos

@ghaerr
Copy link
Owner Author

ghaerr commented May 22, 2023

It would be nice if meminfo can show names of applications like FreeDOS mem.
Is it hard to do that?

It isn't that straightforward, but certainly could be done. I've been wanting the same feature as well for a while. Let me take a look at how we might get this working :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants