Lambda the Ultimate

They explicitly mention the influence of Limbo and Newsqueak in their "Language design FAQ". Btw I misread that initially as "Newspeak" (which is Gilad Bracha's).

Agreed, not mentioning D was quite a disappointment - or rather, mentioning no new major systems programming languages in 10 years in light of D.

... to read or listen to because of the way he doesn't ack prior art

Quite.

Not really been discussed in this thread. Has anyone figured this out yet? I think it is probably the most interesting claim they make that I don't find problematic.

How is defining explicit dependencies interesting? Of course it will help improve compiler throughput - especially if the build engine is smart and ideally implemented using go's channels for coordination among dependencies. Note, also, that "explicit dependencies" being the root improvement of speed is rather misleading.

Apart from that, the Go documentation explicitly states what go source code files the routines came from, which is sort of weird and I haven't figured out yet if that is an option or some weird hardcoding. See: http://golang.org/pkg/go/doc/

Other sorts of "dependencies" information you might be referring to:

http://golang.org/pkg/runtime/

http://golang.org/pkg/reflect/

In terms of how their own packages are organized, it is poor with respect to dependees. There is various formatting utilities scattered everywhere. This does not look like an object-oriented language to me, and in that sense represents about 30 years backwards in evolution. Also, it is strange that they don't have a serialization package where the description of the interface is explicit about the encoding policy, rather than using an interface to define policy -- but perhaps I am missing something. Examples include the malloc package, which does not break away from Inferno/Plan9 and C-style malloc info. Google is basically therefore giving up and saying they will be married to UNIX forever. -- Microsoft solved these problems in 2000 with .NET. Tell me why we are going back 10 years. [Edit: To appease those who dislike .NET, probably the real first implementation of proper serialization was by Atkinson in the '80s for "database programming languages"]

I'm confused. I don't see how taking the address of uninitialized variables is ever a problem. In fact, we do it all the time:

void foo() {
  int x;
  bar(&x);
}

Dereferencing an uninitialized pointer, on the other hand, can lead to serious problems. But maybe I'm missing something?

Code that uses an uninitialized variable as an array index could be a problem because you can use the instruction to corrupt memory. However, in Go, on closer examination, array indexing is bounds checked and it doesn't look possible to cast between pointers and integers. So, after all, my uninitialized variable fear may be baseless.

But "using an uninitialized variable as an array index" is completely different from "taking the address" of that variable. Given a known base pointer p and an uninitialized variable x, using x as an array index is equivalent to *(p+x). We have a problem for sure, but we haven't taken the address of x. Rather, since x is unknown, we can regard p+x as unknown, and then we're back to "dereferencing an uninitialized pointer." So I guess I still don't see what's wrong with taking the address of an uninitialized variable. But maybe it's irrelevant anyway.

Probably irrelevant as you say. Also since every half decent compiler will put out a stern warning against uninitialized stuff - pointers or else. Btw, Go has no pointer arithmetic, so you just can't have expressions like *(p+x).

I thought about posting this link, but then decided that the above mentioned document is almost entirely content free...

Indeed, a lot of the site seems lacking in specifics. After a lot of searching and reading I got a basic idea of the language semantics, but I'm still pretty unclear on some aspects.

So far my impression is that it looks like a decent set of tradeoffs overall, though it's not clear to me why this is a "systems language" in a way that cannot be satisfied by higher-level languages.

I find it interesting that they kept a standard goto. Personally, I'd like a next-gen systems language to support guaranteed tail call elimination, even if tail calls need to be annotated, or a form of safe first-class labels/computed gotos.

The usual question is "why another language." I haven't seen a convincing answer, but maybe it's hidden somewhere.

I don't know if it is convincing, but they do give an answer to that in their Language Design FAQ. It has all the buzzwords including networked and multicore.

The systems level is the one place where I think that question always has a ready answer: as long as C/C++ remain the dominant systems languages, there's always a need. On the other hand, many have tried, but none have succeeded. So the real question for me is: "Why this language?"

Of course. "Why another language" and "why this language" are the same question (if not semantically then pragmatically).

If the competition is Python for internal tools dev, then I see an advantage as the lines between tools and "system code" at Google is a little blurred, and Python has the serious design drawback that the flow-of-control of Python is bound to the flow-of-control of the implementing language (C) - this makes all kinds of things (like concurrency) harder to manage. (Although "stackless python" (not standard) and generators can help.)

Compared to C, it's hard to say -- the .Net CLR, Java VM and newer VMs can all be targeted from multiple languages, but as C has become the "universal medium-level language" (basically a high-level macro-assembler), the spot for "universal garbage-collected human-readable language" that can be the target of source-level code generators (as well as humans), remains open - if this need does in fact actually exist. The inclusion of Goto's I beleive targets this need, as many code-generators use them for efficient encoding of FSNs, etc.

Most of the advantages touted for Go can be had by using other languages with low noise and simply declining to use certain features, whether it be operator overloading or type-hierarchy.

One of the fundamental goals of Go was to be able to compile things really quickly. Lots of things about the language were designed to make it possible to build a really fast compiler, and to be able to do full separate compilation without every needing to re-process anything. (The motivation for this is that at Google, we have...

Google's New Language: Go (blog)

How else should a programming language designer seek fame?

Would Go be getting the level of attention that it's getting today if Rob Pike and Ken Thompson weren't involved? Would it get as much attention if they were not employed by Google?

Isn't really getting that much attention?

Well, Rob Pike's tech talk has over 17,000 views in ~24 hours, already making it the most watched tech talk in the last 6 or 7 months, Google News turns up 83 stories about the language, and it's currently occupying the top two stories on the programming reddit. So yes, I daresay it's getting a fair bit of attention; not as much as V8 and Chrome, but still much more than 99.99% of all new programming languages receive within one day of public availability.

Thanks for digging up these statistics. It is good that I filter the PLT web through LtU not reddit...

Personally, I'd ask "why not Modula-3"? As far as I can tell, they are broadly similar languages, in that they are both garbage collected languages that emphasize static typing, tight control over memory layouts, and overall simplicity and safety.

There are differences of course; Go takes a more modern approach to and has a stronger commitment to concurrency. There is a greater emphasis on tools, especially fast compilation. I haven't examined the OO constructs carefully. It does appear that Go has some very nice enhancements, such as multiple return values, and (someday, presumably) type-safe sums.

Although I will say that the SRC Modula-3 compiler was not too shabby; it turned out very respectable code in respectable time. Turbo Pascal flew rings around it in terms of compilation speed, though the generated code often was not as good.

But Modula-3 was about 5 years too late to compete with C++. Modula-3 was also probably too far ahead of its time, as garbage collection was rather controversial 10-20 years ago. And as Modula-3 was starting to gain traction, Digital Equipment Corporation was failing fast and Java was the hot new thing. So it got upstaged twice.

Other than fame, I would guess that the interest in Go stems from the fact that C and C++ (and Fortran and Ada if you happen to work in the right niche) are the only commonly used imperative languages that are typically implemented via native-code compilation. Honestly, the rise of Perl, Python, and Ruby certainly has helped FP to shed its (mostly undeserved) reputation for being "slow".

currently occupying the top two stories on the programming reddit

Although it should be noted that rather than discussing, say, the semantics or utility of the language, one of those stories is entitled OOHHHH SNAP! "I have already used the name Go for *MY* programming language" - Google Go issue tracker ticket #9.

It seems the interest in the language at this point is primarily of the form "Google has released a language! OMGOMGOMG! EEEEEE!"

Well, that's my contribution to academic discourse for the day.

Go! is a multi-paradigm programming language that is oriented to the needs of programming secure, production quality, agent based applications. It is multi-threaded, strongly typed and higher order (in the functional programming sense).

Go! -- A Multi-paradigm Programming Language for Implementing Multi-threaded Agents

Kind-of interesting in a Mozart/Oz sort-of way.

Naasking ponders "So far my impression is that it looks like a decent set of tradeoffs overall, though it's not clear to me why this is a "systems language" in a way that cannot be satisfied by higher-level languages."

Is it too horribly lame to simply speculate on a why?

First, "systems language" in the community of discourse whence cometh Pike does not always and only mean a language for writing OS kernels. It also includes and can refer to a language for user-space programs. "Systems programming" includes, sure, writing a kernel - but it also includes any suitable low-level, general-purpose, typically close to the basic system capabilities kind of program. The unix programs cron and grep could be considered systems programs. In contrast to systems programming would stand various kinds of very domain-specific programming. Thus, troff is not a systems language, nor is sed, or even (as originally conceived) awk. The latter three would be implemented in a systems programming language but the programming languages they define are not systems programming languages. In general, a language is a systems programming language in that community if its overhead is inconsequential for just about any code you might ever need to write and if it has, at least in principle, access to more or less all of the "essential" features of the machine and the environment.

Second, and this is the purely speculative part: I suspect that somewhere in the vicinity of Pike is a project to build a new OS. My reasoning goes something like this: Consider a massive-scale, Google-style cluster which is dedicated for years at a time to keeping the page cache and indexes and running map-reduce queries. The user-space software on those machines has very narrow capability requirements and is likely highly constrained in what it is permitted to do. At some point you have to count up the overheads imposed by running GNU/Linux on those machines and ponder whether or not there is a significant cost savings (and possible flexibility improvement) to be had by finding a simpler, in some ways more general alternative to GNU/Linux.

What do I mean by "overheads"? One is labor, of course. It is reported that Google maintains a heavily patched, customized GNU/Linux for those tasks. If they are going to have enough workers working on that, could not the same or fewer workers be put to better use on an OS over which Google had greater control? Another overhead is GNU/Linux's "generality" in areas not central to Google's needs on those clusters and possible lack of generality in areas more central to Google's needs. For example, do the really need users and groups on some of those nodes? Do most of their co-routines actually need process protection? How easy is it to cleanly and perfectly allocate one core at of every N as a dedicated incremental garbage collector? Is a POSIX file system really the best choice? Do the network and filesystem stacks really need to be tricked out to be capable of routing so much through user-space? Do you trust a sauna-loving, starting to grey, solo hero not employed by you to continue to supply a steady stream of appropriate kernel updates for your massive computing plants 10 years from now? 15?

Years ago, Pike wrote Systems Software Research is Irrelevant. (That link is to the Freshmeat version but the polemic also exists as a presentation in other forms and places.) Two years later he was hired at Google and you can get a sense of some of his impressions by 2004 from an old Slashdot interview.

I suspect the "... is Irrelevant" paper helped him get the job at Google.

At Google he's confronted with a complete system at massive industrial scale, large (and core business) parts of which are both (a) assembled out of really dubious parts for the task, from a technical perspective; (b) fairly specialized in terms of what they need to do. That bears some resemblance to challenges earlier in his career. For example, there was a time when Bell Labs thought it had invented "personal computing" by which they meant massive time-sharing systems operated by the phone company and a blit terminal in every home.

I suspect that he's "back in his element" at Google and that Go is a systems programming language partly because he and some of his peers intend to write a complete OS for some of Google's subsystems - or at the very least rewrite the user-space for a Linux kernel. (My guess would be the former.)

-t

It's more likely this will be in the vain of Limbo programming language and Inferno OS.

Would you please unpack that a bit ("more likely in the vain of Limbo and Inferno")? I'm aware of Limbo / Inferno and it informed my thinking here. I'm not sure what distinction you are drawing.

Inferno was the OS after Plan 9. It was designed for distributed computing across heterogenous machine architectures.

Sounds like that is right up Google's alley to me.

I am adding the idea of looking at other languages Pike has designed. Limbo was not mentioned anywhere in the thread[Edit: Oops! Missed Allan's first reply to this thread!], and I didn't feel like giving a historical perspective on why it matters. You, on the other hand, took the effort, and I saw it as an opportunity to piggyback on your excellent perspective.

Your cryptically short remark made me think you were arguing against the broad gist of what I was saying when it turns out the opposite is true. Sorry for the misunderstanding.

So far my impression is that it looks like a decent set of tradeoffs overall, though it's not clear to me why this is a "systems language" in a way that cannot be satisfied by higher-level languages.

"Systems programs" means "programs where the constant factors are important". For light-duty, general-purpose programs, the two most important pieces of advice for getting those constant factors down are

1) Avoid chasing pointers, and
2) Keep your multiple cores fed.

Bullet 1 is important because main memory access today is SLOW -- each dereference can stall you for hundreds of cycles. That's why Go doesn't give you reference semantics unless you explicitly ask for it -- if you stick a hundred-element array in a struct, you get an extra hundred fields in your struct. Bullet 2 is why Go has lightweight threading; if you've encouraged programmers to create thousands of threads the odds are that your dozens of cores will all get something to do.

None of this precludes higher-level features; e.g., you can support parametric polymorphism via a size-passing interpretation of polymorphic type arguments. They just haven't implemented it yet. (I bet the Cyclone papers contain a bunch of the necessary techniques.)

If you're really serious about performance, though, you'll need to manage the cache and minimize memory traffic, for which Go offers no help. Interestingly, X10 does do a thorough job here -- there's an explicit notion of "place" to indicate locality properties. It's obviously a good idea; I just wish I knew what it meant logically, on the other end of the Curry-Howard correspondence.

edit: removed, it was an uninteresting comment

"Systems programs" means "programs where the constant factors are important"

I would also add, "precise control over representation", which I think is more general than your point about chasing pointers. As for higher-level languages that support "systems programming", I'll put forth three examples:

1. Felix, which seems to have everything Go has, but stronger typing, parametric polymorphism, and more.
2. The now abandoned BitC, which provides the ultimate level of runtime control.
3. The weakest example is Haskell, but it's legit as there are plenty of low-level DSLs, and the work on House, drivers, and "high-level control over low-level representations".

Each of these projects have achieved what Go purports to achieve with more safety, more performance, and more expressiveness. So I think my original point stands.

It looks like Felix left the SourceForge and moved to a new home.

"Systems programs" means "programs where the constant factors are important".

This is the best definition of that fuzzy phrase that I've ever seen. While I agree with naasking about adding control of representation to the list of factors, that doesn't in any way invalidate the slogan.

I like the BitC literature's observation that systems programming takes place in a context in which representations are prescriptive rather than descriptive. That nailed it for me in a way that no previous attempts at definition had. It also gave me a pivot on which to swing the observation that static typing had previously broken down into two camps: the prescriptivists ("types define how bits are laid out in memory") and the descriptivists ("types define legal operations on terms of that type"), with a lot of people who don't see the value in static typing essentially being prescriptivists, whereas "typeful programming," etc. comes from the descriptivist camp. What's genius—and, so far, as far as I can tell, overlooked—about BitC is that it ends the divorce between prescriptive and descriptive static typing.

Not for nothing, but byte layout orthogonal to type contracts was resolved with the Common Language Runtime, and C++/CLI. Obviously, C++ was an awful language to base this off of, but C++/CLI actually is not murderous to program in like C++ is.

In short, tightly coupling prescriptive (bit lay out) to descriptive (more than just operations on terms) is a bad idea, as these are orthogonal concerns.

For what it's worth, David Harland advocated this approach -- in the '80s! I can probably dig up the reference but it would take a fair bit of effort.

Am I missing your point?

Only in the sense that BitC doesn't tightly couple them.

As for C++/CLI, I haven't worked with it, so I can't comment much, other than to observe that if it's C++, then it doesn't even have let-polymorphism or Hindley-Milner type inference. BitC does, while still supporting in-place mutation and prescriptive representation. As far as I know, those are original contributions.

[[byte layout orthogonal to type contracts was resolved with the Common Language Runtime, and C++/CLI.]]

Why pointing out this particular implementation?
As you say yourself this is nothing new and if memory serves Ada has it natively not with a "Microsoft's extension".

...CLI was only to get Paul to further elaborate on his point, not define a timeline.

neelk: "Systems programs" means "programs where the constant factors are important".

Yes, but not just constant factors. Control over physical representation can let you cut order of time complexity by more than a constant factor. You can use spatial location to imply relationships, giving you constant time operations in places where otherwise they don't occur.

Sometimes I write code with a goal of making all operations constant time (when the population involved scales a lot) and I can't do it without controlling physical representation.

The effective go document is much better in terms of content. It reads as a quick overview of what is unusual in the language, probably what most readers here want to see.

Right, that's a much more useful document. Still unconvinced, though.

nevermind, I see that dynamic dispatch happens when calling methods through interfaces

ken = serious beard
r = no beard
gri = stubble

Correction: gri = clean shaven, at least every time I have seen him.

realloc does very often alloc+copy, virtual memory might provide tricks so it'll "work" fine even when you're approaching the amount of physical ram. In some cases - such as near program startup, or if you ever alloc a few buffers - realloc often does just extend/shrink the piece.
The point is, you have no control over if it has to alloc/copy or just extend and it will do so at leisure (which is why its return value is so important).

Firstly, you should note that Google is a company of about 20k people. I happen to be one of them. It's seldom helpful to generalise about the coding ability of that large a set of people. It's usually more helpful to think in terms of the interests, background and capabilities of particular teams.

Secondly could you point to the pieces of code that have "amateur mistakes" so that I can file bugs and get them fixed.

Thanks

I'm going to go out on a limb and speak entirely inappropriately for LtU with the excuse of bringing this particular subthread to a close. I hope I have enough informal good karma to get away with it.

So, ade: isn't it interesting how you turned a critical observation from a third party about code Google throws over the wall into a solicitation for gratis labor?

If you want to file a bug report, start with that one.

-t

Love Ade's enthusiasm.

I have filed bugs in the past with Google.

I don't have the time to file more bugs with Google, unless you want to create an API where I can submit bug reports via JAX-RS code in a for loop that extends a Java static analysis tool ;-) Just kidding there. ;-)

I know Google is a huge company. I know Microsoft is a huge company. Guess what, Microsoft is four times the size... and I use way more of their products and depend way more on them fixing bug reports. I also know way more Microsoft employees who if I report a bug to them directly, I can be fairly confident it will get fixed within two product releases. My top priority is to make sure our customers report no bugs to us directly.

My major problem with major vendors like Google and MS is always the same: major design decisions out of my hands. Products are open sourced too late for real differences to matter. For example, I was recently overlooking the design of GWT, and how it evolved -- I was considering refactoring it to make it more modular. Some of the early iterations had some design flaws that if you knew a lot about reflection and research in prototyping, you could anticipate, e.g. the introduction of overlay types.

I don't think anyone who isn't Ehud or Anton should speak for LtU. And certainly not to invoke your idiosyncratic opinions about "solicitation for gratis labor".

I didn't take the original message as speaking for LtU. I interpreted it as saying "I know this is out of bounds, please forgive this one time transgression."

Then it was taken as intended. It was snarky and not on topic but a certain (perhaps irrational) button of mine got pushed and so I said what I said.

Thomas: How would you prefer I respond to people indicating that there are bugs in open source libraries I use?

As I see it I have the following options:
- ask the person to tell me what the bugs are so that I can either fix them or file them
- ask the person for the names of the libraries so that I can spend time re-discovering bugs that they've already found
- ignore them

You seem to feel that asking someone to tell me about a bug amounts to asking for free labour. Are you suggesting that I should offer to pay people to file bugs against open source libraries?

I seem to have offended you with my earlier statement so I'm genuinely interested in what you think the correct course of action would have been.

Unless this discussion can be tied to PL issues more directly, I suggest you guys continue it via email.

GNAT.

i.e.,

A comment that starts and ends with “--” is permitted as long as at least one blank follows the initial “--”. Together with the preceding rule, this allows the construction of box comments, as shown in the following example:

               ---------------------------
               -- This is a box comment --
               -- with two text lines.  --
               ---------------------------

This requires the programmer to constantly spend time reformatting the document. Disgusting! This is just as bad as allowing a global variable into a program, in my books.

If you want a block element, use HTML or some rich hypermedia format that supports block element rendering of comments. -- Don't use plaintext and corner yourself into a position of feature envy, because then the biggest problem with your language will be creeping featurism. (This essentially echoes Tim Band's comments on Go's lack of parametric polymorphism, except the context of my comments here applies to gnat, not gofmt.)

Bottomline: Some things in gcc have creeping featurism/feature bloat. If Go only does one thing right, it will be do axe all of this bloat from gcc's toolchain. (Unfortunately, by not supporting rich polymorphism, Go has its own share of creeping featurism, attributable to peculariaties of its designers' biases.)

It does not require the programmer to constantly reformat comments. It strongly suggests developing and using some Emacs customizations to do that automagically.

This is not to dissent from "some things in gcc have creeping featurism/feature bloat". I only mean to point out that editing comments in that format is not necessarily labor intensive.

Do you have a macro I don't know about? ;-)

Here is the scenario I am describing:

               ---------------------------
               -- This is a box comment --
               -- with two text lines.  --
               ---------------------------

Now suppose you want to insert some additional text after the word comment. How do you rejigger that in Emacs so that it automatically wraps what you type to the next line, etc.? What happens when you want to extend the box's width (available typing space) to 74 characters (one space for margin, two for -- on each side) to fit on an 80-column wide terminal?

I only mean to point out that editing comments in that format is not necessarily labor intensive.

The point is that editing a block element should be easy. Bookkeeping should be automatic.

Eliminate labor. Don't manage it.

Now suppose you want to insert some additional text after the word comment. How do you rejigger that in Emacs so that it automatically wraps what you type to the next line, etc.?

You type M-q. There are several elisp packages available for making M-q do the right thing with (different kinds of) box-shaped comments. The oldest one (which won't work any more without some hacking, because it's from Multics Emacs) is from 1979. Sacha Chua recommends her version of Alex Chinn's fairly simple code, and Pymacs comes with a very extensive package mostly written in Python (but nevertheless invocable with a single keystroke in Emacs). Writing a new one (for some custom box comment style that the others don't quite handle) is hardly rocket surgery; you can see that the amount of code required is less than a page.

What happens when you want to extend the box's width (available typing space) to 74 characters (one space for margin, two for -- on each side) to fit on an 80-column wide terminal?

Probably you type M-7 7 C-x f (set the fill-column to 77) but usually I just leave it set to the default of 72.

Bookkeeping should be automatic.

I'm guessing you don't keep the books for a company with any employees, or you wouldn't be saying that. And it sounds like you don't use Emacs much, either, or you wouldn't think this was such a big challenge. Bookkeeping isn't automatic, but refilling box comments is.

"rocket surgery". Nice ;-)

I love using Emacs when it is the right tool for the job, but thanks to WPF, I find it is getting easier to write custom visual DSLs that help me tackle problems. Most of my problems can no longer be solved by buffered editors. They're inherently aided by visual programming. -- I've never been aware of that macro, probably because I've always outlawed such commenting style as seeming to be needless extra maintenance.

I prefer to focus on writing code, code, code.

Not choosing what drapes to decorate the windows with.

Bookkeeping should be automatic. If you want to work with the accounting analogy, I'm fine with that. We have lots of legacy accounting customers from the FORTRAN packages we sold in the 70s and 80s and early 90s. I can tell you that the biggest human-induced flaw in this process is poor bookkeeping. At the end of the reconciliation period, you are supposed to "close the books". Yet, many clients can't even do this step correctly. Closing the books should be an automatic operation, but due to incorrect operating procedures in these accounting departments, they might take 1-3 days to close the books. This is a systemic failure of bookkeeping.

Some bookkeeping requires human intervention. However, you should strive to limit that bookkeeping to the beginning and end of a long chain of automated processes. This limits potential for error to a few distinct places. For example, a general ledger.

Bottom line: I want this to be easy, you want it to be possible.

why not Haskell-style Layout? if we're interested in programmer productivity and removing unnecessary furniture (such as brackets round if predicates), why stick with mandatory braces?

I've never seen a Haskell program that keeps all the optional braces and semicolons, and most don't have any. On the other hand, many ECMAscripters seem to love putting in the optional semicolons...

Historically there would be weird bugs when different browsers inserted the optional semi-colons in different places. Those who have been bitten by this in the distant past have learned to always insert the semi-colons themselves.

For example: http://robertnyman.com/2008/10/16/beware-of-javascript-semicolon-insertion/

FWIW, Haskell's rules are more robust, so bugs like the one shown in that blog cannot happen.

I'm sure that helps avoid endless discussions about the correct layout.

Wishful thinking. What happens when your definition of "bug" differs from the whimsical opinion of Pike and Thompson?

With Go we take an unusual approach and let the machine take care of most formatting issues. A program, gofmt, reads a Go program and emits the source in a standard style of indentation and vertical alignment, retaining and if necessary reformatting comments. If you want to know how to handle some new layout situation, run gofmt; if the answer doesn't seem right, fix the program (or file a bug), don't work around it.

This is why tools like FxCop, StyleCop and Mono.Gendarme are essentially rules engines. You simply distribute the rules external to the engine and its toolchain. Now when Gendarme's developers fix a bug in the engine, you don't need to go through a complicated patching process. You simply pull the new sources from upstream, compile, and use your existing configuration settings. Note: bugs can also include performance enhancements, or memory leak fixes, not just parsing/lexxing correctness!

"Are there true systems programming languages (i.e., ones suitable for implementing serious operating systems) that are garbage collected?"

Yes, sort of. Apple's version of Objective C has garbage collection, and Apple's OS is largely written in it. But garbage collection was only added recently, and I doubt that much of the OS uses it, so far.

Talking of Objective C, we should add that to the list of systems languages that Go is competing with. And possibly also Nu, which is a LISP based on Apple's (or GNUStep's) Objective C class library (http://programming.nu).

Mac OS X's Mach kernel and the BSD subsystem is written in C/C++, not Objective-C. Only framework libraries are written in Objective-C.

Er, ONLY the framework libraries? ONLY?

Framework libraries aren't part of the OS.

A lot of the framework libraries are written in C or C++, too. Run nm on them. The OS X framework libraries that are written in Objective-C don't depend on the garbage collector, because it's still optional, and they have to work in programs that don't use it. Instead they use reference-counting and autorelease pools.

Apple's version of Objective C has garbage collection, and Apple's OS is largely written in it. But garbage collection was only added recently, and I doubt that much of the OS uses it, so far.

In this sense, C++ is a garbage-collected language, since it is quite easy to have a garbage collector look for unreferenced, non-destructed objects. In fact, you can do the same, with a bit more fuss, in C. But this isn't at all what I mean. A serious PL for writing an OS in seems to (i) need the ability to carefully manage the locality of data structures, and (ii) be able to keep a low profile when writing critical sections of code,ensuring the code does exactly what is says and nothing more, like, for instance, suddenly invoking GC.

LispM-like projects are interesting, because they change the landscape of what the OS expects from the language. It might be that the future of OSs lies with LispM-like breaks from UNIX-like OSs. But that looks to me to be betting against the weight of OS practice.

Right. In these senses, the new capabilities in Objective C are as yet untested (I assume - I haven't tried to check, and anyway the relevant code is closed source).

Objective C 2.0 is open source(APSL) and available at opensource.apple.com. The Objective-C Release Notes[RTF] details what the changes are for 2.0 when it first appeared in Mac OS X 10.5 Leopard.

Perhaps that is the idea?

new Chromium OS comes out, doesent allow executibles, but what if you could compile large programs in a second or two by pointint the browser at google's cloud?

GC isn't nearly as costly on a platform where it properly integrates with virtual memory.

Wishful thinking.

They may not report long pause times, but that is simply because they lack test cases that reveal these data points exist.

Interpolation and extrapolation from the data points in that paper doesn't cut it. - just ask the folks at Azul Systems that provide hardware-accelerated GC.

What's so fanciful about the thesis that GC touching pages which have potentially been paged out causes page thrashing, and hence significantly reduces performance?

Read Azul's whitepapers on hardware-accelerated GC.

It reframes the problem you are pitching me. "Paging" is so irrelevant here. For desktop PCs, sure. For embedded devices, that cool hack is probably not relevant there either.

The same guys who founded Azul founded 3DFX.

Cameron Purdy had an excellent blog post about this recently, although he didn't discuss Azul at all or what business ventures are the big players here.

If your server is paging anything out, you're probably doing something wrong, because your application is in a much better position to choose when to do disk I/O than your OS is. (The Varnish guy disagrees, but the people I know who run Varnish in production give it enough RAM so it doesn't need to page.)

Don't they only run GC on OOM errors?

I asked below "Are there true systems programming languages (i.e., ones suitable for implementing serious operating systems) that are garbage collected?" I guess I have to qualify what I mean by "serious", since Lisp Machines were certainly serious, but I think that not all design choices made by them have stood the test of time.

If I change the question to: Are there true systems programming languages (i.e., ones suitable for implementing serious, high-performance operating systems) that are garbage collected? — then I think that excludes Lisp Machines.

I didn't work on lisp machines but I had friends who did and I've seen various documentation over the years.

My impression is that low-level OS functionality was coded in styles that mostly avoided ordinary consing: some parts wouldn't cons at all, other parts might use explicitly managed memory pools - regular consing only where it was clear that it wouldn't cause performance problems.

The infamous situation of filling memory and having the machine essentially become comatose for N hours while collecting memory was triggered when applications consed too much, too quickly.

Furthermore, that seems to have been a limitation from the hardware and GC technology of the day. Lisp machines were, in modern parlance, "single core". While incremental GC techniques with small upper bounds on pause time were known, their overhead on such hardware would have been painfully high. The solutions to very long pauses were to either reboot the machine instead, or change the applications to cons less. I'm not sure that's any worse (or even much different) from applications on, say, a unix machine that overflow physical memory and page thrash (which doesn't make C a non-serious or non-high-performance systems language).

How good or bad was the Lisp Machine's garbage collection anyway?

Lisp Machines implemented Henry Baker's incremental GC pretty much from the beginning. But it was of dubious usefulness at first, because it had to scan all of memory on each collection cycle, causing paging activity. By 1983 (maybe earlier, I don't recall exactly) the GC was also generational, which made it far more usable. Really, it worked very well, at least for applications that had the expected behavior which generational GC exploits, that most objects die young.

Ok. The application some of my friends were having "fun" with was running, in the late 1980s, on some TI Explorer model. It was, for its time, a very large scale search program trying to maximize the score of tiling various surfaces with various shapes according to constraints. I don't know how exactly they blew out the GC (perhaps they normally ran with it switched off?) but I do remember them reporting that periodically they had to reboot rather than let the machine embark on a GC pause (or perhaps just extreme slow-down) that would take several hours to complete.

What was going on there was quite common and is still standard practice for users of most implementations of Lisp and other languages with GC. The Explorer definitely had an incremental, generational collector, which your friends almost certainly were using. But with a straight generational collector, the objects that survive repeated collection cycles eventually get "tenured" into the oldest generation, which with time can get much larger than main memory. At that point the choice is between a nonincremental collection of the whole heap, which as you say can take hours, or restarting the process (which on the LispM meant rebooting) and reloading whatever data are still needed.

Again, that choice still confronts anyone doing long-running computation in most garbage-collected language implementations. The notable exception is the JVM (Sun's, certainly, and probably IBM's as well), which uses Hudson and Moss's "train algorithm" to efficiently collect longer-lived objects. I don't know of any Common Lisp implementation (excluding ABCL, obviously) that uses the train algorithm or anything like it, so we CL users are pretty much stuck with this for now.

I don't know for sure, but I would guess that the situation is similar in the ML/Haskell world. I think a generational collector is required for any serious implementation, but I would be surprised if anyone had gone to the trouble of adding the train algorithm. Similarly for Scheme. (I'd guess the CLR would have it, though.)

In the mid-80s we shipped Lisp Machine (TI Explorer) systems for on-line applications, and chose to have them do a nightly reboot. It was much faster than a full CG which would have been required otherwise.

While incremental GC techniques with small upper bounds on pause time were known, their overhead on such hardware would have been painfully high.

This is not correct. The incremental GC was implemented in microcode and the overhead was quite reasonable -- until one filled main memory and started paging. See my previous post, below.

Don't use the past tense about LispMs: there are still a few in use!

The most special feature of LispMs is that there is one memory management service provided by the OS that all applications are expected to use. I am guessing that this is necessary for GC-on-OOM to be workable. I'm pretty sure that no new system programming language will be widely adopted if it makes this design choice (i.e., be tied to this particular design choice in the OS)

Don't use the past tense about LispMs: there are still a few in use!

LOL, I suppose there would be. I still have an XL1200 that worked the last time I fired it up, but that was a couple of years ago. Are you still using one?

Are you still using one?.

Hah. I've seen them, but never used one. I keep seeing mentions of LispM activity here and there. I have seen an ITS machine in use, too.

If anyone has a working LispM that they wish to unload, they know how to make a generous addition to my understanding of many things.

I've had an eBay saved search for a lisp machine for awhile now.

I bought a DataHand keyboard for $1100 a year ago to add to my museum of awesome nerd crap.

Quite. But unless we know what's going on, or can make educated guesses based on the design of the language (unlikely), not really on-topic for LtU (though I have been known to speculate here on a google DSL from time to time...)

Bell Labs invented "personal computing" and planned it as a distributed network of unix timesharing systems with a BLIT terminal in every home. The leap from there to Google data centers and thin-ish client terminals is not a large one. In any event, regardless of that, Google's unique data centers are a unique and unprecedented super computer, inviting new systems software rather than merely duct tape, ad hoc assemblies of systems software aimed at smaller scale super-computers.

This might not be exactly what you intended by "to see what GHC would do with similar code," but there's some other Haskell examples here:

http://www.reddit.com/r/programming/comments/a4n7s/stackless_python_outperforms_googles_go/c0ftumi

That whole Stackless Python discussion concerns me a bit, but it brings up some interesting comparisons.

Here's a Haskell version by Don Stewart using MVar instead of Chan. It outperforms the Go version in most cases.

Visual Haskell! Suspected that actually. Compiling with ghc 6.10.4 gives acceptable performance, though not linearly scalable with number of threads. Don's post mentions the use of 6.12 with tweaks, which is not released yet, but is necessary for linear scaling.

Btw mvars or chans doesn't seem to matter for the benchmark. Also the eager evaluation $! in don's code doesn't seem to make much of a diff as well.

Well, let's rather say that the culprit is the version of GHC integrated in VH. GHC has made huge progress in the stability of performance improvement by parallelism and concurrency lately, especially with GHC 6.12 and the trend will probably continue with the next few versions.

GHC is steadily becoming a tool of choice for parallel programming, it already had most of the APIs (MVars, Chan, STM, Strategies, DPH ... though Data Parallel Haskell is still in development) and it is now being fine-tuned to allow reliable use of them.

I didn't mean "culprit" in an incriminating sense ... just "hey VH, you don't have the latest cool stuff in GHC" :)

but, again, it's more fair to compare Go to Rob Pike's last language, Limbo.

Well, that's a fairly generic description that you quote.

And half of it does not apply to Go anyway...