Modern Shell Language

Due to a project I'm working on, I need a general-purpose command language. That is, a language optimized for writing ~80 character programs that mostly call external functions, piped together in an interesting way. Some things, like type-directed parsing, seem like a good way to improve upon POSIX sh, but I'm wondering what else I'm missing that would be nice. Currently on my list of things to try:
  • Type-directed parsing
  • Unifying piping and arguments (some sort of "stream" abstraction, maybe)
  • Semi-structured content (structured content isn't great for very short programs, since the user is actually looking at a decent chunk of the outputs; but semi-structured content means less endless reparsing)
I've also been toying with replacing linear piping with a general graph structure, but that seems far too complex to be worth it. I'm hoping to aleviate the need for a full-fledged language by allowing easy scripting (in JS), so that narrows down the problem domain significantly...

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

PowerShell and Vistrails

PowerShell is obviously related. I remember reading a very interesting document about the PowerShell design from a programming language point of view. All that I could find is this less on-topic message from a developer. Z'Bo certainly knows much more about this.

Your mention of a "general graph structure" made me think of "workflow systems". I'm quite ignorant of these and other will probably come up with better references, but I learned about them in this presentation of VisTrails, a tool meant for flexible scientific computation and visualization workflows. Note that this is using a graphic language rather than a textual language; my intuition is that, for not-too-complicated not-too-automatizable situations, representing graph visually is a better user interface than a label-based textual system, but might also be less productive for one-shot small workflows.

Some related LtU threads (some other threads may be relevant as well, but didn't generate any discussion):

PowerShell is structured around RAD

The PowerGUI IDE for PowerShell is the best "visual shell". Unfortunately, pash, the Mono *NIX implementation, has mostly been abandoned. So PowerShell is really limited to Windows for anything useful.

There are also some annoying bugs with the PowerShell interpreter. PSv1 code may not run in PSv2, due to ambiguous operator overloading resolution rules. Also, the module system in v2 isn't much of a module system, and there isn't something like Ruby gems or CPAN and a tool to download them. Also, it uses dynamic scoping rather than a more formal system for passing around tramp parameters.

Tangent: Why don't shell languages that use dynamic scoping just use lexical scoping and have a technique for passing around tramp parameters? I think Simon Peyton-Jones experimented with that in Haskell, but don't know much about the outcome. But isn't it 'better'?

Edit: It appears 'tramp parameter' is the non-academic word from this. Google says I took that phrase from Code Complete. Academics like Erik Meijer call it 'implicit parameters with static types'.

The Scheme shell; Es

scsh

Scsh has two main components: a process notation for running programs and setting up pipelines and redirections, and a complete syscall library for low-level access to the operating system

Es: A shell with higher-order functions

If this is for writing

If this is for writing scripts that interact with the complicated logic of a server (daemon) or simulator or some such, you can add some glue logic to make its functions callable from an scripting language (python, perl, scheme, tcl, ...). That way you don't have to reinvent everything + you can take advantage of existing libraries, dbms etc, should you need them. For example, you can write a cgi script to drive a web interface.

For interactive use a simple CLI is easier to use but that can be built in the scripting language. A friend once did this to aid hardware engineers in their debugging. He added functions to peek/poke hardware registers + other low level hacks, to mzscheme. He then wrote a simple REPL in Scheme so that users didn't have to type the outer most parentheses + a bit more for more convenient syntax. This was good enough for the engineers. And they didn't mind typing e.g. (+ x y) to add two numbers -- this was more than the CLIs they were used to provided!

Now if you are just using this project as an excuse to write the shell you always wanted, you can ignore the above!

The strength of the /bin/sh ecosystem is that it is easy to tie together most program in interesting ways since they accept and produce plain text. If you replace text with typed or structured data, you'd have to reimplement all the standard tools. Still, if using standard tools is not relevant, you can probably use s-expr as the format of communication.

Handling general graphs in a linear language is painful. If you go down that path, consider a GUI. Providing an easy to use scriptable GUI can be an interesting project by itself.

Trees

While general graphs can be difficult to use, bash goes a long way with trees. Take a look at what you can do with subshells and redirection.

Things to address

I have a new shell language deep down in my TODO list as well. Personally, I think that (ba)sh got more wrongs than rights, but somehow it's still fantastically productive to work with, even for non-interactive use. The problem is to fix those flaws and still have a script language that still works well...

Here's some of the things that I want to fix:

  • static typing
    • extensive type inference
    • extensive implicit type conversions
  • avoid tokenizing errors (like spaces in file names)
  • error handling
    • actually have error handling :)
  • unit testing
    • lots of issues here
  • module systems, get away from the old source / . Needs to support partial imports and type compatibility/inference on the API level.

I also have a dream to get away from VT220 terminals (but still have a terminal), and away from the monospace fixed height fonts/layout and syntax limited to US-ASCII... would still require a vtXXX emulation layer, and no matter how artificial the term is, the source files would end up as "binary" as opposed to "text"... I think I'll file this dream together with the "shift the earth axis" one...

reacting to asynchronous events

This is a subject I love to muse about. But as of yet, very little action :) A common affliction to visitors of this place I guess.

Anyhow, one thing I seem to be missing in your list, is event oriented programming. In contrast to streams, which have been ``solved'' in unix/sh, reacting to (asynchronous) events is quite a bit harder.
A use case would be the launching multiple jobs in parallel, with graceful handling of failures.

Could you expand on ``type directed parsing''?

Are you aware of termkit? It's a really interesting attempt to improve the shell using modern UI.

Almost interesting

Shok

A shok session is a voyage along two parallel modes: the command-line and its companion programming language.

[..]

A { at the start of a command-line swaps you into a block of code.

shok:~ : {
shok:~ >     new x = "Cool!"
shok:~ >     print("This is a code block. " ~ x)
shok:~ > }
This is a code block. Cool!

[..]

In code, use : to make a single command-line statement:

shok:~ : {
shok:~ >     : mkdir foo
shok:~ >     : echo "hello, world!"
shok:~ > }
hello, world!