Lambda the Ultimate

I would have thought it fairly obvious that explicit parallelism is based upon targetting a known number of independent CPU threads. But with all the attention now aiming at massively increasing the number of cores and independent threads on each chip then the known number will only be appropriate for a limited snapshot of time.

I don't see how this is relevant. Explicit parallelism is programmer-directed concurrency, meaning the programmer is spawning a process explicitly for a reason that can't necessarily be known to the compiler. For instance, a shared webhost would explicitly spawn a process for a hosted customer application, so it can be killed if it misbehaves. It's just one example that has little to do with exploiting the maximum concurrency.

Further, virtual processors are reified as vproc data types, so the program can scale itself to whatever number of vprocs are available.

So why does GC halt all other processing when it takes place?

It doesn't. Most collections are local to a vproc, and only rare global collections block other vprocs from proceeding. There are known ways around this limitation, but they went for a simple approach for the first iteration I guess.

Should this be determined explicitly by the programmer who uses domain knowledge that each calculation is relatively lengthy, or implicitly by the run time system dynamically adjusting its strategy based upon the available processing capacity?

That is the distinction between explicit and truly implicit parallelisation.

Your examples highlight the reason why they added parallelism hints in the language itself (parallel arrays and tuples, etc.), but not why explicit concurrency constructs like processes and channels are necessary; explicit concurrency can certainly be used to solve the same problems though.

My example highlights a clear-cut case for explicitly managed concurrency constructs, since there's no way to kill an implicit thread/terminate an implicitly concurrent computation early in general.

As to your other point, the compiler and/or runtime doesn't always know enough to be able to automatically split the processing load, or determine when something is fully parallelizable, which is the motivation for parallelism hints.

About mapping a function across a list (rather than an array) -- task queues are natural for that sort of situation, though they are more useful when the amount of work per list element (or group of list elements, perhaps) is enough to justify creating a task. Creating a task is different than creating a thread -- threads are relatively heavyweight, so it's more natural to have a fixed pool of workers which grab tasks.

I think of lists as streams -- naturally sequential objects that require transformation in order to be parallelized by data (rather than by task, via a task queue). Arrays are naturally parallel objects, in contrast. So it's not really natural to abstract away the details of a collection data type in the parallel setting; you have to know if something is "array-like" or "list-like."

I agree that in general there's no way to kill an implicit thread, but does it matter?

I'd say yes, because processes can be non-terminating. A language is an operating system of sorts, and as an OS, I want to be able to orchestrate execution of various entities and isolate them if necessary.

I suppose that my argument boils down to always presuming parallelisation unless something has been explicitly sequenced, (e.g. by monad laws)

What are the consequences for the developer's mental model? Sequenced semantics are simple to reason about (or at least, are well-known).

You appear to be arguing for the reverse in that you are looking for explicit concurrency primitives, thereby implying that the default behaviour is sequential processing.

I just want some framework where both are available. I know explicit concurrency primitives are necessary, as evidenced by my example (or at least some form of external termination control -- see Orc's filter expression for example); I want to be able to use this control when necessary, but otherwise, I'd love it if the compiler could parallelize everything it can!

I wonder how explicit and implicit concurrency intermix as pertaining to vprocs, etc. For instance, suppose I launch a process whose expression has many parallel subexpressions. What does the scheduling trace look like for each vproc? Do the sub-computations inherit the process's slice? I'll have to read their schedulers section more carefully, because I didn't really grasp it on the first read-through.

At least for scientific computing, we have two or three generations of programmers trained to assume sequential semantics of mathematical expressions. The latter may include operations on vectors or collections which have implicit parallel semantics, but the mathematics already suggests this kind of collection-oriented meaning. However, if I write y = A * x (for a matrix A and vector x) and then afterwards write w = y' * y, I expect an implicit sequencing. Furthermore, it's not clear how making these operations implicitly parallel would help performance, in the usual situations. Reasoning about how to decouple them is something I'm doing now for a PhD thesis; I don't expect compilers to do it and I don't _want_ them to, because it changes the rounding error significantly.

I see implicit dataflow parallel semantics as something that might be useful in some domains, but not in all.