Lambda the Ultimate

I don't think either syntax-case or SRFI 72 can be declared more state-of-the-art than the other. SRFI 72 makes a number of design decisions which diverge from those made in syntax-case. One more obvious example is the use of lists (more or less) in SRFI 72 to represent syntax, as opposed to the [opaque] abstract syntax objects used by syntax-case. Take a look at the SRFI 72 discussion, including threads such as Opaque syntax objects, loss of abstraction, and The power of Lists for some flavor of the opinions on just this one point. There are quite a few other such choices, none of which make either system more "state-of-the-art" relative to the other.

A number of these design decisions, including the syntax representation, could in theory be changed in either system, i.e. they're choices which aren't inextricably bound to other aspects of the nature of the host macro system. There are a host of intermediate points between the original syntax-case and SRFI 72, and other points outside that space. Syntax-case itself occupies a few different points in the space, e.g. the original system and the system in PLT Scheme have some notable differences.

The R6RS macro system is unlikely to match any of these systems exactly. Aside from the purely technical factors, there are also factors to consider such as backwards compatibility with existing Scheme code (which is likely to favor syntax-case), the amount of experience with the macro system (which favors syntax-case), and ease of implementation (which may favor some of the choices in SRFI 72). In addition, R6RS will still allow for implementations to extend the language, which might (just as an example) favor the choice of abstract syntax objects over lists.

In any case, the R6RS editors will certainly be considering the viable options.

Thanks for the overview. I'm only left with one question. I haven't looked too deeply at SRFI 72, but I recall that part of the motivation for that system is a perceived flaw in the hygiene of prior systems. Is this also controversial, or is this kind of fine-grained capture control simply not an issue enough of the time?

I apologize if this stuff has been discussed to death on comp.lang.scheme or the SRFI lists (I guess I need to start following those). I've been pleased with syntax-case, and would be quite satisfied if R6RS standardized it.

I haven't looked too deeply at SRFI 72, but I recall that part of the motivation for that system is a perceived flaw in the hygiene of prior systems. Is this also controversial, or is this kind of fine-grained capture control simply not an issue enough of the time?

That issue is real, at least with respect to syntax-case. (It doesn't apply to e.g. syntax-rules, because it only affects macros which depend on syntax-generating procedural code which introduces identifiers.) I'm not sure that there's any major impediment to adopting the SRFI 72 "modified hygiene rule" in a version of syntax-case, though, which would eliminate the problem.

The srfi-83 library forms looks similar to s48's configuration language, with additional bells and whistles (and no file form...). One of the complaints about this configuration language, as I recall, is that it was supposed to be incompatible with syntax-case, yet now we see the macro system is to be syntax-case like. I'm puzzled as to how this is resolved. Is it something to do with those indirect-exports?

Which similarities are you thinking of? In the sense that there's a distinct and fixed module definition language which can be reliably extracted from the code of a module without much computational effort (e.g. it can't be obscured by macros), the SRFI 83 and Scheme 48 systems are similar. However, the SRFI 83 system is closer to PLT's modules both in the specifics of its (smaller) feature set, and in the sense that SRFI 83 modules must include the actual code of the module being defined, whereas S48's system is geared towards separating the module definition from the code.

I'm not sure that either of these things have a bearing on the issue of compatibility with syntax-case — I recall having seen a claim of incompatibility, but I don't recall the specifics. Googling turns up an observation by Taylor Campbell in this message:

you can't just tack psyntax onto any old Scheme system and expect it to work; hygiene and macro expansion plays a great role in constructing an AST from an S-expression, and so it's often very tied to the implementation. psyntax works in some implementations because they let psyntax do all that work for them (for instance, SISC does this), but those that have their _own_ mechanism for this will have problems using psyntax. For instance, Scheme48 doesn't support SYNTAX-CASE; it would be a _huge_ kludge to make psyntax work in Scheme48

However, that's referring to the fact that the portable syntax-case implementation, psyntax, can't just be dropped in to Scheme 48 — it would have to be integrated at a lower level, effectively requiring a new implementation. This is similar to the case with PLT Scheme, which has its own implementation of syntax-case, integrated with its module system.

BTW, Andre van Tonder has an implementation of the R6RS library specification in terms of his SRFI 72 macro system, here. That macro system includes an implementation of syntax-case.

Maybe I'm missing some of the bigger issues

I'll attempt to describe some of those.

SRFI 57 is basically a purely syntactic system with generative records, which is roughly equivalent to the "Explicit-Naming Syntactic Layer" in SRFI 76. If that's the only functionality you want, you'll be able to import just that functionality as a separate library. (SRFI 76 doesn't mention libraries, because it was released before the library SRFI — this is a work in progress.)

If the R6RS only specifies such a syntactic layer, Scheme implementations will need to extend their record support in various ways, e.g. to add reflection capabilities to support printers and inspectors, and to provide a more concise and implicit definition syntax for teaching and scripting contexts. This would result in a partly standardized record mechanism, which in practice would be different in every implementation.

Ordinary SRFIs don't have this issue: you can use SRFI 9 or SRFI 57 to get records, but it's typically integrated into the implementation's underlying record mechanism, which already specifies many of these things, so that e.g. an IDE's record inspector will just work. SRFI 76 specifies not only the functionality in SRFI 9 or 57, but also the infrastructure that a realistic implementation of those SRFIs would depend on. Aside from ensuring standardized record support across implementations, this will also help support the writing of portable third-party tools.

These different sets of requirements are addressed in SRFI 76 via four compatible but fairly independent components, as summarized in the Abstract and Rationale sections of the SRFI. These components will almost certainly correspond to separate libraries in R6RS. For the most common uses of records, a programmer would only need to be concerned with one of those libraries.

That's one take on the bigger picture. As for the details, there's room for discussion there — I'd suggest that anyone interested get involved in the SRFI discussion, which is still open.

Okay, I see what you mean. I look forward to seeing how all of this is resolved -- and what the various Scheme implementations will do.

To me, the first paragraph of the introduction section of R5RS says that Scheme is an exercise in designing a language consisting of a minimal number of mutually orthogonal features.

As Carter has pointed out, there are also other principles involved. I'd like to address this point a little. A nice example of minimality in R5RS Scheme is call/cc, which can be used to implement all sorts of control flow mechanisms, including exception handling mechanisms. However, that doesn't help a portable library raise an exception which can be understood by arbitrary client code. Some standardization is needed for that purpose, which is why the R6RS will specify an exception handling mechanism. So although the minimality principle is the first guiding principle listed in the latest R6RS Status Report, there will be areas, particularly in libraries as opposed to the core language, where it is not the primary relevant principle.

Re records specifically, merely implementing "records of fields indexed by names" can easily be achieved with pure R5RS, as the various SRFI reference implementations demonstrate, so there's no need for additional primitive features to support that capability alone. Instead, the R6RS records proposal is largely about specifying a standard syntax and semantics for Scheme records, i.e. it falls mostly on the standardization side of the the minimality vs. standardization divide.

An important part of that standardization is providing the ability to define enforceable abstractions, both for the benefit of the programmer as well as for the Scheme implementor who doesn't wish to be constrained to implementing records in a particular way. The enforceable abstractions requirement argues for ensuring that records are only accessible through their defined accessors and mutators, etc., which implies (more or less) that defining a new record also defines a new abstract and disjoint "type". Efficiency also can't be ignored here: one of the R6RS guiding principles is to "allow implementations to generate efficient code", and requiring that implementations expose the innards of their records as ordinary data types is at odds with that.

How much of an issue some of this is depends on what alternatives you have in mind. To discuss this further, I think we'd need to look at more specific proposals.

For example, if you're thinking of something like the Perl approach of allowing arbitrary values to be "blessed" as belonging to a certain type, that would raise concerns about abstraction safety, static reasoning, and efficiency. If you want to do that sort of thing in Scheme, nothing stops you, but the host Scheme implementation doesn't really need to be involved in any special way — you're essentially designing your own type implementation mechanism. Something similar is true if all you want is an Erlang-like mechanism where records correspond to tuples, and are neither opaque nor a disjoint data type.

[...] merely implementing "records of fields indexed by names" can easily be achieved with pure R5RS,

Indeed. What you can't do in R5RS is to make new "types". Unless you represent records as procedures, all implementation details will leak through, and even with a procedural representation the fact that records are procedures will be quite visible.

what alternatives you have in mind

What I have in mind is to have a facility for extending the set of builtin "types". Perhaps one possible way to do it would be to have a facility that allows one to generate fresh "keys" and use them to "lock" (and "unlock") arbitrary objects. By default, a "lock(ed) object" would be completely opaque. To create abstract types (with a set of procedures to manipulate them) one could restrict access to keys lexically. Such a facility should also allow one to register various auxiliary procedures for each new key, such as procedures for displaying and comparing locked objects.

One question is what precedent there is for a feature like this in existing Schemes. The R6RS process is not primarily about inventing new features for Scheme, but rather standardizing features that will "allow programmers to create and distribute substantial programs and libraries". Most such features are already fairly well established in existing Scheme implementations, and if they aren't, it raises questions about how much experience we have with them, whether they're appropriate for Scheme, etc.

A feature such as you describe could be added in a future RnRS, and the record system's generation of types could then be explained in terms of that feature. I don't see a strong reason to address such a feature in R6RS, given the current goals.

It could make for a nice SRFI, though!

I put together a library for extremely minimalist typed boxes with only one procedure, NEW-TYPE: it takes no arguments, creates a novel box type, and returns four values, all procedures. The first procedure creates boxes of the new type; the second is a type-test predicate for the new box type; the third provides access to the contents of any box created by the first procedure; the fourth takes an argument which becomes the new contents of such a box. One can provide arbitrary amounts of security over such boxes by choosing which procedures to expose and which not to.

For single inheritance, one could extend this to return a fifth procedure value, which is isomorphic to NEW-TYPE except that the box types it creates are subtypes (in the sense of the test predicates) to the newly created box type.

I think it's a little unfair. They seem to be restricting themselves to areas that cause real pain when trying to write portable Scheme programs. Records, modules, exceptions, syntax-case... these are things that virtually every R5RS implementation provides, and virtually every substantial program uses. The differences between implementations are irritating at best, and in some cases positively confounding. And Unicode is simply too important to ignore.

When these features can be defined in terms of a simpler core language, so much the better! Standard ML has core and derived forms, and I've never heard anyone suggest that the derived forms don't belong in the standard. Of course, R5RS already has this distinction as well! Obviously this is a balancing act between minimality and standardization, but I don't see too much evidence that R6RS is headed into the weeds.

And speaking of portability pain, if I had one small wish for R6RS, I'd like to see a standardized sort function, even if it's not perfect. A simple "sort" function taking an (optional?) comparator and a list/vector would be a blessing... Of course, this probably belongs in one of these new-fangled "libraries"... ;)

Your sorting wish echoes Olin Shivers' comments in SRFI 32 ("Sort Libraries"):

"It's just amazing to me that in 2002, sorting and merging hasn't been
completely put to bed."

I won't quote the rest - Olin is a good ranter. Some of it might be less true now, for some implementations, but of course there still isn't anything standard (and SRFI 32 is withdrawn).

SRFI 67 ("Compare Procedures") could make for a solid foundation for a sorting specification, though.

Haha. At one point I read almost the entire text of SRFI 32, getting myself more and more worked up, before noticing that it had been withdrawn. Then I banged my head on my desk for about ten minutes...

I understand Olin's goals (and they are noble as always) but it seems like SRFI 32 must have been a little too ambitious. Lack of a full standardized library of sort routines shouldn't prevent us from getting a single "sort" function for simple jobs...

Some people are using SRFI 32, e.g. there's Will M. Farr's implementation for Bigloo. The sort library in Scsh 0.6.6 is from SRFI 32, according to the release notes:

"The old package SORT from Scheme 48 has been replaced by a sophisticated library written by Olin Shivers for the withdrawn SRFI 32."

The decision to withdraw a SRFI is formally speaking the author's, and says nothing about whether the facility may or should be implemented. There are plenty of implementations of withdrawn SRFIs out there.

First, I'd like to point out that I plan to personally proofread the R6RS, so it'd be a "mandatory" procedure. :oP

Anyway, Barak, now that we've gotten you to delurk, what would you put in [or take out of] the R6RS?

The question is: what niche should Scheme target? The niche it currently dominates is: language that is so simple its basics can be learned in an hour and an interpreter written in a page or two. This niche is great for teaching, and for prototyping new language features and implementation technologies.

By adding goop, we make Scheme less suitable for this---the niche it dominates. That is a boneheaded move. If we want a fancy Scheme standard for writing big programs with a zillion standardised bells and whistles, we should call it something else. Like "Industrial Scheme." And we could derive it be starting with Common Lisp and removing what we don't like (dual namespaces, symbol structures, etc), refactoring the rest, and changing naming conventions.

But we should preserve a small elegant language actually useful for teaching and for rapid implementation. This means: no records, no error system, no standardised XML parsing libraries, none of that. Back to basics. Back to the powerful features needed to rapidly write an interpreter, or to easily prototype a new kind of specializer. I guess what people sometimes call "core Scheme."

(I'd personally even be tempted to remove call/cc, since experience has shown that it is brittle in the face of multithreading and therefore as a community we desperately need to reconsider the design space for dealing with continuations. A reconsideration that has been greatly hampered by the ---in retrospect--- premature standardisation of call/cc.)

I've heard this argument several times, but don't understand what you stand to lose, if Scheme evolves in a direction which makes it suitable for other things as well. Won't the R5RS version of Scheme, or even the IEEE standard, continue to meet your requirements?
And for teaching purposes, doesn't PLT Scheme demonstrate well how different language levels can be provided for teaching purposes, without limiting the evolution and growth of the language? An R5R6 compatibility mode is planned, if I recall correctly, for R6R5. The "language" feature of the module system provides this capability.

Some of us (many?) want to use Scheme for "real work" and think it has potential to compete with Perl, Python, Ruby and, how knows, maybe even Java. But only if it provides, as a minimum, the stuff planned for R6R5. Even with modules, records, exceptions and Unicode, it still will be difficult going without a standard object system.

You suggest Common Lisp should/could be a starting point. One needs to ask why it hasn't been more successful and whether RSR6 can learn any lessons from Common Lisp's mistakes. For one thing, Common Lisp seems to have stagnated. The SRFI process in the Scheme community may be helping to give Scheme some momentum.

But we should preserve a small elegant language actually useful for teaching and for rapid implementation. This means: no records, no error system, no standardised XML parsing libraries, none of that. Back to basics. Back to the powerful features needed to rapidly write an interpreter, or to easily prototype a new kind of specializer. I guess what people sometimes call "core Scheme."

The R6RS should provide exactly what you've just described. Not much detail is provided in the status report, but it does mention that the R6RS will specify a core language and set of separate libraries. This should meet your specification: "preserve a small elegant language actually useful for teaching and for rapid implementation."

One of the guiding principles published in the recent status report is "allow educators to use the language to teach programming effectively, at various levels and with a variety of pedagogical approaches". The core/library split is part of the support for that principle. As you know, Scheme is well-suited to this sort of separation and factoring, since even fundamental syntax (e.g. LET) can be defined in a library.

Another principle is "allow researchers to use the language to explore the design, implementation, and semantics of programming languages." I don't see any serious barriers to the R6RS being able to implement both of these principles.

Some of the specific concerns you've raised make for good illustrative examples. The R6RS will not specify a standardised XML parsing library, because no language-level support is needed to make such a library work. There are already some excellent XML parsing libraries for Scheme, including portable ones such as SSAX and SXML. In many cases, a good portable library is better than a standard, because it serves a need well without necessarily reducing choice. So we'd like to encourage portable libraries.

Accordingly, the focus in the R6RS is not on actually specifying libraries such as SSAX and SXML, but rather on making it easier to write and maintain such portable libraries. Making SSAX and SXML as portable as they are, and porting them to the various Scheme implementations they run on, required a significant effort, many aspects of which are ongoing as versions change. The R6RS should allow such libraries to be implemented in pure portable R6RS Scheme, which will have a strong positive effect on the availability of portable libraries for Scheme.

Besides, even if the R6RS did specify an XML parsing library, if it is specified as a library, how would that affect the use of Scheme for pedagogical and research purposes? If you're not interested in a particular library, don't pay any attention to it. This applies to records just as well as it does to XML.

The only potentially real issue I see here is that of features whose presence might complicate the core language. The exception system might be one such feature, but all realistic implementations of Scheme contain an exception system. The R6RS will simply specify some requirements that such a system should conform to. For some idea of what this is likely to look like, take a look at SRFI 34. It's nice and short, and should not be an onerous requirement for an implementation.

If you want to implement a toy Scheme interpreter in a page or two and leave out the exception system, I see no reason that the R6RS will stop you from doing that. You could never implement a full R5RS Scheme in a page or two, either. If anything, the core/library split, and the factoring of libraries, will make it possible to be more explicit about what you are implementing with respect to the R6RS.

With C++ or Java, students often learn just a small island of
constructs carved out by the instructor within an enormous ocean of
mystery. When they look at the manual, this view is reinforced. They
learn that the semantics of a construct can be determined only by
trial-and-error, and that there is no point trying to actually learn
the whole thing.

One big advantage of Scheme is that students can learn the whole
thing. Or at least it used to be, because the new R6RS (which they
will read, and which will burble up to them in its full complexity via
system code, error messages, etc) has all sorts of extraneous dross in
it. Like records, which directly duplicate functionality already
available in *two* other ways: lists and closures. The module system
is another hunk of overwhelming complexity.

Make no mistake: adding things to the standard has a cost, not only in
making Scheme more brittle and harder to use as an implementation
testbed, but also in cognitive load for students. The standards
committee should be looking for things that can be *removed* in light
of improvements in implementation technologies, and pedagogical
problems, rather than things to add.

If set-car! and set-cdr! had been removed, then boxes would have had a stronger justification. Otherwise, there doesn't seem to be a strong argument for them in the Scheme context, and as you say, you can easily define them with the record system.