Streaming Support

[mipearson]

cc @sj26 @keithpitt

The problem (as described to me by @keithpitt) is terminal (and its ecosystem) need to support the common use case of being able to see output as it happens, including being able to display in-browser those clever npm/yarn/docker spinners and progress bars and so forth.

Keith and I discussed a few ideas:

Move the whole thing to JavaScript

BK sends the browser raw ANSI, the browser parses it and displays it to the user. Bor-ing.

terminal parses the ANSI, the browser interprets it

Terminal is responsible for parsing the ANSI and turning it in to a set of instructions (eg left; up; color 3; write bob) that are then sent to the browser for interpretation.

terminal parses the ANSI and renders it, sends browser individual lines

Terminal parses the ANSI, renders it to an internal buffer, and at a set interval (once per second? faster? when a certain amount of input is read?) sends the lines to the browser. The browser receives the lines and either adds them to the DOM (if it's a new line) or replaces it (if it's an existing line).

Terminal would stream JSON blobs (with one blob per newline) that would look something like this:

{ line: 1, html: "<span class=..." }
...
{ line: 50, html: "Loading.. 10%" }
{ line: 50, html: "Loading.. 20%" }
{ line: 50, html: "Loading.. 30%" }

[mipearson]

Personally, I prefer option 3, but if I was worried about moving parts and so forth I'd be strongly considering option 1. I have some sentimental attachment to this project but it's mainly an excuse for me to hack on something in Go where performance is a factor more than anything else.

If you went with option 2 or 3, you could integrate that directly in to the agent to reduce the load on your own servers, and have the agent send both ANSI and the line changes. Also, less moving parts & service management. That said .. we (Marketplacer) very rarely ever upgrade our agents (because lazy/risk averse) so backward compat may be an issue for some of your customers.

[ticky]

I’d been thinking we’d go for option 1, but I think I’m having the engineer’s fallacy of seeing an interesting problem I want to take on when there’s already a viable solution in place. 😅

I figure as much as it’d be nice to defer this responsibility to the Agent, it might well make more sense to defer it to the end user’s computer if anything. I think the possibility of pushing the raw ANSI over a websocket came up at one point.

Now the interesting thing is that these options seem to flout the existing line limits we have in place. That’s potentially okay if we’re buffering things correctly and ensuring the lines are accepted correctly, but it’d be worth double checking our math when we calculate which lines are valid.

[mipearson]

but I think I’m having the engineer’s fallacy of seeing an interesting problem I want to take on

I have exactly this problem ;)

[mipearson]

Hello!

I'm looking for something to hack on this RailsCamp, and this seems appropriate as it's Go and Javascript.

Did you end up reaching a decision?

[mipearson]

At RailsCamp I worked on a proof of concept for streaming support: https://github.com/buildkite/terminal/compare/mp/Streaming?expand=1

Demo by building it then running cat fixtures/npm.sh.raw | ./terminal-to-html -rateLimit 100 and opening http://localhost:6060

Supports multiple clients (had it up to 25 at RC during the demo). If you stop the server and re-run it with new input it'll refresh also.