Implementation of various pipes

[raklaptudirm]

This discussion is for deciding on how to implement pipes in mash. Here are the topics for discussion:

• Details of how pipes will be implemented in code.
• Settling on a proper syntax for piping stdout and\or stderr.
• Whether to add syntactic sugar for writing/appending to files.
• Details of how the output and exit-code of a command may be stored in a variable.

[raklaptudirm]

Proposal

Details of how pipes will be implemented in code

Piped commands will be executed parallelly, and the stdout of the source command will be set to a buffer, and the same buffer will be set as the stdin of the destination command.

// grep -r "todo: " . | do stuff
var buf bytes.Buffer

// grep -r "todo: " .
grep := exec.Command("grep", "-r", "todo: ", ".")
grep.Stdin = os.Stdin
grep.Stdout = buf
grep.Stderr = buf

// do stuff
do := exec.Command("do", "stuff")
do.Stdin = buf
do.Stdout = os.Stdout
do.Stderr = os.Stderr

grep.Start()
do.Start()

---

Settling on a proper syntax for piping stdout and\or stderr

????

---

Whether to add syntactic sugar for writing/appending to files

Previously, the decision was to declare commands that could do that:

$ grep -r "todo: " . | write todo.txt

I propose to add some syntactic sugar to do that instead, which will work exactly like the above:

$ grep -r "todo: " . > todo.txt # or >> todo.txt

This will also allow us to write output to variables:

$ grep -r "todo: " . > {todo}

---

Details of how the output and exit-code of a command may be stored in a variable

????

[tjgurwara99]

Ok, this seem to be the same discussions point I had raised when I was reviewing the PR earlier which related to the order of operations. I remember that you wanted to do this using write but I disagreed at the time - still do.

Now, just to clarify a few things:

You want to run the pipes in parallel by default? From my understanding, pipes can never really be parallelised since they by definition depend on the output or one command as input for the other command.
Currently all shells use the & operator to achieve that - I personally think that is good enough - its simple for people to parallelise their code if they want to yet be deterministic by default so as not to make it difficult for programmers to understand what went wrong.

Syntax is a question of taste - its your program, you can design it however you want. However, I think it is better to rely on symbols rather than letters (makes it easy to lex properly)

Writing + appending to files shouldn't have a command in my opinion. The reason being that when we will be running processes (commands) in mash, we will have full control over the streams (in/out/err) which means that you should be able to write the them without the presence of a command as long as the symbols are distinct. (> and >> are clear on their meaning - you can have something similar.)

About storing output as variables, what do you mean exactly? Do you mean something like the following?

SOMETHING=`ls -l`

That already stores the values properly and I had mentioned this before that the command vs not command route would have problems with variables since you have the var or let keyword in the beginning to indicate the declaration of a variable.

If you want to store the output and exit code into a variable, I think a custom type Output that can store the exit code along with in/out/err together in the memory would be useful - which means that people can store a command and run it with different inputs on the fly since the type itself would have additional information but that means you need to have a notion of a type in there as well. But this is just a simple suggestion (not well thought out yet - so take it with a grain of salt.)

[tjgurwara99]

😂😂😂😂 nvm I understand what you meant now. I read that sentence as "exists" instead of "exit". Ignore my dislexic self🤣

But again going back to the parallelism point, the problem would still persist. The pipes in Unix are very specific, the yes command continuously sends y\n to the stdout which is the stdin of the other command, because of the pipe. If you make them parallel, there is no guarantee of the order in which these pipes access the stream.

Essentially, (from my understanding) it would create a race condition on these stdin and stdout. Because in a piped command

a | b

The stdout of command a is equal to the stdin of command b. You can dig through the open source bash/zsh implementations to see this. So my concern mostly is about why you are thinking of making it parallel because there are actually other ways that bash and zsh handle this - using the & operator where they don't share any streams which means there won't be any race conditions.

[raklaptudirm]

I am trying to understand how something like yes | head -n10 would work if the shell is not running the commands parallelly. Here is the output from my machine:

ψ yes | head -n10
y
y
y
y
y
y
y
y
y
y

[exit 0]

If the head command has to wait for yes to finish, then this job would run infinitely, as yes runs forever. Also, I do not understand how a race condition would come up, as the buffer.Buffer essentially acts like a channel, where in the above example yes is sending the data and head is receiving it.

Also instead of a buffer.Buffer, we could use (*Cmd).StdoutPipe() as the stdin to the next command.

We can also see in this example that the command whose output is being piped is run concurrently using cmd.Start(), which is equivalent to starting a goroutine.

[tjgurwara99]

If the head command has to wait for yes to finish, then this job would run infinitely, as yes runs forever. Also, I do not understand how a race condition would come up, as the buffer.Buffer essentially acts like a channel, where in the above example yes is sending the data and head is receiving it.

This perfectly sums up everything. But what you're talking about is concurrency not parallelism. This is why I was concerned with the idea of parallelism because suppose command a | b again and further suppose that b can take no input as well as take stdin then if b runs first, there stdin is empty so it would not be deterministic what would happen. If things are running in parallel then there is no deterministic order of execution of commands - which command would run first is unknown. Whereas the kind of parallelism (concurrency) you're talking about has always been in shells which is not something new here so sure we can work with that - buffers have always been concurrent so it's not difficult IMO. In fact, we should support bash style pipes completely.

[raklaptudirm]

Since we are on the same page now, I want to extend the go interpretation of the pipes:

command 1 | command 2 | command 3

r1, w1 := io.Pipe() // pipe from cmd1.Stdout and Stderr to cmd2.Stdin
r2, w2 := io.Pipe() // pipe from cmd2.Stdout and Stderr to cmd3.Stdin

// command 1
command1 := exec.Command("command", "1")
command1.Stdin = os.Stdin
command1.Stdout = w1 // pipe to command 2
command1.Stderr = w1 // pipe to command 2

// command 2
command2 := exec.Command("command", "2")
command2.Stdin = r1  // pipe from command 1
command2.Stdout = w2 // pipe to command 3
command2.Stderr = w2 // pipe to command 3

// command 3
command3 := exec.Command("command", "3")
command3.Stdin = r2  // pipe from command 2
command3.Stdout = os.Stdout
command3.Stderr = os.Stderr

// command 1 handler
go func() {
  command1.Run()

  // close pipe
  r1.Close()
  w1.Close()
}

// command 2 handler
go func() {
  command2.Run()
  command1.Process.Kill() // kill source command 1

  // close pipe
  r2.Close()
  w2.Close()
}()

// command 3 handler
command3.Run()
command2.Process.Kill() // kill source command 2

The code seems to work as expected. Any opinions?

[tjgurwara99]

No opinions, do what you'd prefer - though you're forcing parallelisation on it again (not concurrency) but I'll leave that to you to figure it out if you think it works.

Also, these days I'm quite busy so I won't be able to contribute to this project for a significant foreseeable future; so I think you can revoke my write access on it for the time being 😄