Lambda the Ultimate

That would be interesting to see in a modern imperative language.

Actually, I believe it to be quite doable. I wasn't handling 'verification', but I did explore the implementation of the whole restart mechanisms here.

Considering how symmetrical that proposed solution is to the existing systems, and that I can't find any implementation hurdles, I'm more surprised that we don't have it already.

It can't be more irritating than Java's checked exceptions.

That has me seriously laughing out loud. Java developers should know better than to integrate 'new' features in what is intended to be a mainstream language without seeing them tested first.

In any case, checked 'resumption' conditions could be made part of the exception or part of function signatures, but I'm not certain it would be a worthwhile pursuit as checking these things would ultimately limit user-created resumption policies. Actually, I'm against checked exceptions for the same reason: they severely limit the distance for which error handling policy can be decoupled from the code that introduces the error.

Actually, I'm against checked exceptions for the same reason: they severely limit the distance for which error handling policy can be decoupled from the code that introduces the error.

I don't think that's so much a knock against checked exceptions, as it is a knock against the policy that exceptions can't propagate automatically to enclosing scopes. I think every function should have an effect variable which denotes the exceptions that can be thrown from its execution. This effect variable is a union of all effects of calling child functions (which should be fully inferrable). 'main' should simply have a signature with a nil effect, so no exceptions escape the execution of the program as a whole. Thus we achieve checked exceptions, but without the headaches that Java imposes.

Ah, yes. That would be correct; it isn't the 'checking' that is the problem, but rather both the requirement for manifest declarations of what must be checked and the inability to automatically propagate such checks. In what I wrote above, I was considering "checked exceptions" with the Java design.

Regardless, I'd be tempted to simply guarantee all exceptions are checked by having the 'default case' for exceptions be provided as a standard behavior via the process or thread task, allowing programmers to override these defaults at will.

This effect variable is a union of all effects of calling child functions (which should be fully inferrable).

I have a hard time imagining how this solution would work at module boundaries without sacrificing separate compilation.

On the other hand, I'm not so opposed to sacrificing separate compilation so long as one can at least integrate some pre-compiled forms (with 'pre-compiled' being less restrictive than 'separately compiled' in that it allows one to have a compilation ordering requirement).

'main' should simply have a signature with a nil effect, so no exceptions escape the execution of the program as a whole

There are times it would be quite useful for something like 'main' to propagate exceptions so long as the host knows how to process them.

This is OT, but the whole prescribed notion of 'main' and 'program as a whole' is something I dislike as a language standard concept for other reasons. How a host environment (such as Unix or a shell) interprets a library or dictionary of executable or evaluable code should really be left outside of the language's definition.

I have a hard time imagining how this solution would work at module boundaries without sacrificing separate compilation.

Since every function now sports an effect variable, and ML modules can be statically erased, I don't see a problem in principle. First-class modules or modules as first-class values might be more challenging since the module used could change at runtime.

How a host environment (such as Unix or a shell) interprets a library or dictionary of executable or evaluable code should really be left outside of the language's definition.

I disagree. I believe it was you who was arguing that languages and operating systems should be essentially unified, and I agree. In that case, consider a system where the OS is built in a safe language, ala Singularity OS. The user's shell dynamically loads the code for the program, but in order for it to launch the program, the program entry point must have a well-defined signature, such as implementing some Application interface:

interface IApplication {
  void Main(string args);
}

The signature could be arbitrary of course, but there has to be something well-defined for the dynamic code loader to typecheck and the shell or OS to invoke.

The user's shell dynamically loads the code for the program, but in order for it to launch the program, the program entry point must have a well-defined signature, such as implementing some Application interface:

Modulo reflection, I agree. But I'd note it important that it is the the user's shell that defines "the necessary interface". The language designers are not 'prescribing' any particular meaning to 'IApplication' or 'Main'. To the language, those are not special at all.

With reflection - the ability to inspect the 'object' file selected for execution - things become much more interesting. A shell could use 'main' by default, but still allow access to other functions via simple parametric extensions to the name (such as ':foo arg1 arg2 arg3'), with (' args') being the same as (':main args').

Access to type-descriptors (foo : int int -> int) for the arguments could be used to help parse the arguments or even support tab-completions, and could certainly be utilized to support typesafe shell operations and workflows.

Alternatively, a shell could default to 'open ' as the behavior, automatically importing both 'foo' and 'main' and any other exports as commands in the local environment and thus allow one to divide the execution from the environment manipulation. Or both could be possible (in Java, for example, one may use classes without importing them, so long as they use the long name). In this sense, the shell is really an interpreter for a given language that can be extended, and also happens to (in the default configuration) include extensions to help locate commands a user might be interested in executing (such as installed services).

#myshell> open math;
ok
#myshell> square 20
400
#myishell> x ## (square x) == 81
x=9
#myshell> play funnymovie.avi
executing play:main filename=funnymovie.avi...

I believe operating systems, shells, and languages should be fully unified. I would not stop at half-measures like forcing everything through a shell's 'IApplication' interface.

This is probably getting a bit off topic for the thread, but Windows PowerShell might be of interest to you. It's a shell which, amongst other things, allows .NET objects to be piped between commands instead of just text; and which specifies an interface that objects can implement for better integration with the command line (including argument-specific tab-completion).

But I'd note it important that it is the the user's shell that defines "the necessary interface". The language designers are not 'prescribing' any particular meaning to 'IApplication' or 'Main'.

Other than it's the application initialization point, I don't see what kind of meaning there could be.

Access to type-descriptors (foo : int int -> int) for the arguments could be used to help parse the arguments or even support tab-completions, and could certainly be utilized to support typesafe shell operations and workflows.

I don't think reflection adds anything really. One can just as easily create a strongly typed interface for all of the features you're describing, and the application can itself provide the tab-completion.

public interface IApplication {
  void Main(string[] args);
  string[] InferArg(string fragment);
}

I think concise language constructs, like lambda args and type inference, help more than reflection here.

Other than it's the application initialization point, I don't see what kind of meaning there could be.

Despite that you can't see it being used any other way, can you offer a good reason why the language designers should prescribe such a notion as 'Main' being an 'application initialization point'? Why should language designers even accept the concept of 'application'? You seem stuck on the idea of reinventing a Unix shell in a language. That is not at all the only option available to you.

I don't think reflection adds anything really.

Reflection buys you the ability to implement the shell as a common language with a shared parser used by all the applications. It also buys you the ability to implement strongly typed workflow languages (like pipes and filters, or service mashups).

One can just as easily create a strongly typed interface for all of the features you're describing

Strongly typed, perhaps. But you're going to be performing runtime typechecking in any shell application because the 'code' isn't produced until runtime. Runtime/dynamic typechecking (weak or not) may be implemented inside 'Main' by reinventing a poor-man's parser for each and every application. Alternatively, you can invent it once and only once (as well as unify syntax for running commands) by having both the typechecking and the parsing duties be moved into the shell.

and the application can itself provide the tab-completion [...] I think concise language constructs, like lambda args and type inference, help more than reflection here.

The approach you propose is considerably less flexible or "strongly typed" than it seems you believe it is, considering that you'll be unable to support composition of commands and functions and you'll still suffer type mismatches whenever you need to parse input parameters (except worse because YOU need to implement the parsing and the parameter validation each time). At the same time you lose efficiency for both the programmer (who must integrate a parameter parser into each application) and the runtime (which will be continuously marshalling and demarshalling structured data as strings).

I'm all for static typing whatever can be typed statically. But providing static typing over commands parsed and introduced at runtime means knowing types. And knowing types of objects that have already been compiled into programs requires the same set of environmental features as to support runtime reflection.

Reflection buys you the ability to implement the shell as a common language with a shared parser used by all the applications. It also buys you the ability to implement strongly typed workflow languages (like pipes and filters, or service mashups).

Reflection buys you far more dangers than benefits, and I used to be firmly in the other camp. Of course, this all depends on the scope of the reflective capabilities, whether it includes abstraction-breaking introspection, etc.

And reflection doesn't buy you any workflow or filtering capabilities. Those are readily implemented by higher-order functions and/or objects, as evidenced by any ML, none of which have reflection.

I think what you really want is to embed the parser and a language interpreter, ala "typing dynamic typing". So you basically want a REPL shell.

But you're going to be performing runtime typechecking in any shell application because the 'code' isn't produced until runtime.

Not necessarily. Code isn't loaded until runtime, which is very different from saying that it's produced/generated at runtime. Either way, a program must provide a type signature for the code it is attempting to load:

val load_code: 'a Sig.t → in_channel → 'a Rep.t option

Rep.t is a 'representation type' often appearing in polytypic programming literature (which is a safer form of reflection). For simplicity, I'm assuming the Sig.t defines the structure of the expected type.

Now the user's shell is a text-based interface, so we can go two routes here: a full blown REPL shell, or a smaller standardized interface language for constructing and invoking subsystems.

You also seem to imply that my original suggestion required the entire program to be loaded into memory, but really, only a small launcher component need be instantiated for tab-completion and initialization. A language-based OS is a sea of objects, not heavy monolithic processes like current OSs, so when I say IApplication, I mean literally a single object, not all the objects instantiated when Main is invoked.

[Edit: note that, even with the REPL shell option, the code must still export either a standardized symbol table, or a standard entry point which is executed on code load, which loads all symbols into a global table. There always a standard interface somewhere! :-)]

And reflection doesn't buy you any workflow or filtering capabilities. Those are readily implemented by higher-order functions and/or objects, as evidenced by any ML, none of which have reflection.

'IApplication' is "a smaller standardized interface language for constructing and invoking subsystems." It is not ML. It does not have ML's capabilities.

By using 'IApplication', the shell program cannot determine the 'real' types of the inputs to the published applications. That is, when a user types in a 'wrong' string, the shell can't issue any warnings or do anything similar. The shell certainly cannot typecheck its workflows or pass complex objects as inputs to 'application' methods.

I think what you really want is to embed the parser and a language interpreter, ala "typing dynamic typing". So you basically want a REPL shell.

Essentially, yes. But it doesn't need to be a REPL shell; the HCI is more open, and could be an Object Browser akin to Squeak, ToonTalk, or Second Life. But a REPL browser would be a convenient version.

And the typing doesn't need to be dynamic. But static typing of each shell command, as it arrives, requires the language runtime maintain all the information one would usually associate with static type reflection.

The ability to create alternative shells that were not strictly REPL loops would certainly benefit from the ability to query for the appropriate metadata, implying reflection.

Reflection buys you far more dangers than benefits

Eh, well, I'll agree it can cause problems. But the issue is never reflection by itself, but rather reflection in the context of some assumptions used in the development of other language features. And I'm willing to trade some of those assumptions in order to buy reflection without the dangers. Besides, most of the assumptions 'problematic' when combined with reflection (such as use of abstraction as a mechanism for encapsulation, and the use of encapsulation as a mechanism for security, and the use of named types with inheritance) are also problematic for other language features that may be nice to have in an OS-integrated language (including distribution or security).

Code isn't loaded until runtime, which is very different from saying that it's produced/generated at runtime.

I literally meant produced/generated at runtime. But you might be confused: I'm talking about the shell code - the stuff the user types dutifully into the REPL interpreter or whatever. The shell code is among the stuff that, if "the language and shell were integrated" could be typecheked, abstractable, capable of producing and handling exceptions, etc.

You also seem to imply that my original suggestion required the entire program to be loaded into memory, but really, only a small launcher component need be instantiated for tab-completion and initialization. A language-based OS is a sea of objects, not heavy monolithic processes like current OSs, so when I say IApplication, I mean literally a single object, not all the objects instantiated when Main is invoked.

What you say here confirms what I already understood of your claims. I don't take issue with the above. Where I have a problem is you presenting 'IApplication' as a solution then pointing at 'ML' when attempting to explain the properties of your solution.

The link I provided earlier to tagless interpreters and "typing dynamic typing" can be used to achieve what you're looking for. A language's standard library can provide a module which is a self-interpreter for the language (or a JIT preferably), and standard library can provide another module for the language's parser. Constructing a REPL shell from this is somewhat trivial. I think all languages should be built in this metacircular fashion in fact; metacircular interpreters for statically typed languages FTW! :-)