Lambda the Ultimate

Reasons for such a thing.

* Debugging; many systems allow a function to inspect the call stack, which does provide information (often with types erased) about calling objects and contexts.
* As a crude means of implementing things like dynamic scoping (in a dynamically typed context) or access control (if (caller != approvedCaller) throw AccessException; and stuff like that).
* Some aynchronous messaging protocols provide this by default, mainly to ease construction of a return message.
* Because it's available. :)

Note that I'm not endorsing either; but they are there.

Reasons against:

* As noted above, it increases coupling between caller and callee.
* Security issues--if a caller is to invoked untrusted code, it might want some guarantee that the callee won't inspect (or modify!) its state inappropriately.
* There are generally better ways to do whatever it is you're trying to do--the caller can always be passed in explicitly. Some of the "better ways" require explicit language support (visibility restrictions in OO languages, explicit dynamic scoping, etc)

I'd figure that some sort of reflection would be the obvious way to get information about the call stack (ignoring the problems presented by TCO).

Anyhow, aren't many of the listed pros and cons applicable to reflection in general?

Indeed, those pros and cons apply to reflection in general. And note just how weak those 'pros' really are! weak privacy properties, a wide boundary for covert channels, poor support for a flawed security model, an inefficient memory model for deep and modular functional and procedural composition, and unnecessary coupling including (in the above case) a binding of messages to a "call stack" which harms concurrency. What's not to love?

Reflection is one of those design-bugs that people like to call 'features'... it's like a little baby dragon, all soft and cute and lovable while playing with toy programs. Then your programs grow larger, distributed, extensively modular, and it bites your arms off.

Dynamic scoping is a useful property. Make some form of it explicit. Allow the whole dynamic scope to be saved and restored, such that thunks can be associated with the dynamic scope in which they were created rather than in which they were executed, and to allow scope barriers be introduced (e.g. between the caller and a script interpreter). Make dynamic scope 'immutable' such that the continuation changes its scope rather than modifies the scope others are using, as this is good for concurrency and distribution. Put the 'System' information in dynamic scope, i.e. for logging errors and reporting to 'the console', thereby allowing system information to be changed. If you must use something like ACLs, then have a sender earlier in the call chain inject a challenge-reply object into the dynamic scope (which can respond to a challenge with a signature).

But avoid reflection! or at least limit its scope. For scalability, for security, for privacy: the callee shouldn't be able to "reflect" on the caller, and vice versa - the caller shouldn't be able to see the implementation-details of the callee, not even for resumable exceptions. (Put resumption and other error-handling policies in dynamic scope, too, if necessary.)

If you haven't already, you should read the paper referenced in this LtU post and the paper passingly mentioned on LtU here. Probably the latter first, then the former. But yes, I certainly agree that reflection shouldn't be something anyone can do any time they feel like.

The explicit argument approach has the benefit of enabling delegation. The immediate most calling code might be part of some code generator boilerplate code, or a proxy of some sort.

What it sounds like you want is an answer to "whose message is this?" A common answer is to have authors sign their posts. Otherwise you might blame the messenger.

Something that can look at itself is NOT a design bug.

Even objects, as a default, carrying a pointer to 'self' is problematic for partial-evaluation and object-graph rewriting optimizations. The ability for a program to look at itself ultimately ties the implementation and memory structure of the program to the source structure.

I don't understand how you tie reflection as a general concept to large, distributed, modular systems.

I agree that some limited-scope forms of reflection are unlikely to be problematic. It's really the ability for a module to have a non-local impact or make non-local observations - including non-parametric observations on inputs to the module from outside modules (i.e. asking for the class of an input given by interface) - that spells trouble for security, distribution, and scalability.

Problem is, I see reflection as nearly being defined by the violation of parametricity. "Reflection" is the ability to take an arbitrary program unit (even one that you didn't create) and reflect upon it - see how it ticks under the hood, or what it's made of. Use of reflection is to make decisions based on this observation, which introduces the problematic coupling between one component and the implementation of whatever other component created the object you're observing.

The mere ability to represent (i.e. quote) code as data, and later interpret it (in a possibly modified language, or via a macro system) is not, by itself, reflection. Reflection is a form of meta-programming, but not all forms of meta-programming involve reflection.

Most of the design flaws in Maude have nothing to do with reflection whatsoever (e.g., Maude I/O) as far as I can see.

I haven't used Maude, and my only experience with a similar language (CafeOBJ) was a semester in university on automated theorem proving. I, therefore, can't speak on it overmuch. But I can suggest that the flaws of reflection could possibly be masked by the flaws with I/O, or perhaps reflection is challenge that makes inter-module I/O more complex and has thus delayed effective integration.

Problems with reflection begin when there is communication between modules or components, and use of reflection on that communication.

supercompilation brings up a good point, because one of Turchin's principles of metasystem transition was that the system ought to be simple enough to analyze itself. Thus, reflective systems should be remarkably simple and easy to understand.

Sure, a system should be as simple as possible (and no simpler). But your conclusion doesn't follow. The ability for a system to analyze itself doesn't imply the ability to analyze itself from within itself. That just introduces a new and interesting problems of non-termination and such (ask and reflect on: "will the answer to this question be no?").

Reflection is part of any system with reflection. When a system with reflection analyzes itself, it would also analyze the analysis of itself, and the analysis of that analysis of itself, and so on.

Reflection is a poorly defined term, or perhaps more accurately, solutions requiring reflection tend to indicate a poorly specified problem. It sometimes indicates a type of abstraction that does not yet have a typed equivalent. That set is much smaller than it once was, with well known approaches for overloading/ad-hoc polymorphism, polytypism, extensible records and variants, first-class labels, open data types and open functions, and polyvariadic functions, there's not much advantage left for real reflection IMO.

Reflection is part of any system with reflection. When a system with reflection analyzes itself, it would also analyze the analysis of itself, and the analysis of that analysis of itself, and so on.

When reflection research on Lisp systems was hot in the 80s, this was called the "reflective tower" IIRC.

Reflection is a poorly defined term

Right down there with 'meta-programming', eh? You won't hear an argument from me claiming it to be well defined. I can only say that my own understanding of 'reflection' as distinct from other 'meta-programming' techniques is that reflection happens to involve observing (and potentially modifying) the runtime structure of a program in a manner that rapaciously violates encapsulation.

I place reflection near meta-object protocol on the list of damning 'features' that harm the mature-language upper limit for optimization, scalability, security, flexibility, safety, and other nice properties. Every language has inherent limits to how effective it will be in full maturity, based on decisions made early in design (not just for the core language; common libraries matter, too). Reflection and meta-object protocol each do much to lower that glass ceiling.

I'm much more fond of macros or extensible grammars, partial evaluation, scatter-gather code styles (like rules-based and logic programming), and staging as meta-programming techniques.

solutions requiring reflection tend to indicate a poorly specified problem. It sometimes indicates a type of abstraction that does not yet have a typed equivalent.

Poorly specified problems don't require reflection, or much anything other than better specification. The use of reflection in those circumstances probably a consequence of exploratory programming. If the language didn't have reflection, but instead had some other feature unnecessary to the solution or problem, you'd probably see that other feature being used instead.

The latter is more common - the use of meta-programming techniques available to a language in order to work around language limitations or weaknesses. That's one of the big reasons for meta-programming, after all... providing the tools to Greenspun whatever you need.

But when meta-programming is used on the grand (non-local) scale, I believe it suggests a deficiency that must be fixed in future generations of the core language...

Something still missing from many languages is the ability to gather code and data scattered across many source files and combine it in an ad-hoc manner without side-effects. (I.e. you can do it in Lisp or Scheme with side-effects and mutable state, but that introduces ordering issues and such should any two constructed objects depend on one another.) Support for this in the language would probably avoid huge classes of code-walker meta-programs and preprocessors.

The sort of "well-known approaches" you list would also qualify, but it's worth noting that several of them (open data types & open functions, along with various classes of polymorphism, for example) are simply limited forms of the above 'provide, gather, combine' pattern. Too many techniques tackling the same problem demands a generalization of the technique.

this was called the "reflective tower" IIRC

You probably recall correctly. I saw the term "reflective tower" in Chapters 1 of the Maude 2 manual, plus the document Reflection in Rewriting Logic and its Applications in the Maude Language (1997), looked up when reviewing John's comments.

I suspect the excitement about these 'reflective towers' was about as relevant to productivity and utility of languages as the excitement over discovering that a Turing Complete language can emulate any other...

I'm guessing you have some kind of highly dynamic, syntax changed out from under me view of meta-object protocol? Because in my view a good meta-object protocol is a typed, sane version of macros.

I'm guessing you have some kind of highly dynamic, syntax changed out from under me view of meta-object protocol?

Maybe so. I've seen meta-object systems in Ruby, Smalltalk, Lisp, and Python, each of which are decidedly 'dynamic' languages.

I also accept as axiom that modules and other forms of code-you-don't-control will be needed in any non-toy project, such that it's politically impossible to discipline all code used by a project. This tends to have me thinking that the ability for programmers to achieve predictable properties under deep composition of grey-box code is among the top features for a programming language.

a good meta-object protocol is a typed, sane version of macros

And where can I find this alleged "good" meta-object protocol?

Meta-object protocol tends to apply globally, i.e. modifying all instances of an object of a given class, or even all classes. It is difficult to ensure these modifications 'compose'. Modules using the meta-object protocol can easily conflict, or break modules that weren't using it. Worse, sometimes they do compose but it depends on hidden details such as the loading order for the modules, so you might get two modules working correctly but then they break due to some tweak in seemingly unrelated code. And that's even before considering security for mobile code and such.

Macros apply locally, to a region of source-code. So long as you avoid code-walking macros (which are problematic), macros are modular. Hygienic macros can also be defined, of course, but even non-hygienic macros are safe (supposing the environment in which the macros are defined is safe). Macros may or may not be typed; it doesn't really matter if they are typed, so long as the IDE can report errors in terms of the macro prior to expansion.

I'm curious as to in which capacity you believe a meta-object protocol is a "typed, sane version of macros".

And where can I find this alleged "good" meta-object protocol?

I don't know - I should have warned in advance that my feelings on this stem from my own work.

Meta-object protocol tends to apply globally, i.e. modifying all instances of an object of a given class, or even all classes.

Yes, this is what I meant by highly dynamic MOP - I agree this is a bad aspect of e.g. the CL MOP.

I'm curious as to in which capacity you believe a meta-object protocol is a "typed, sane version of macros".

Because the values computed by macros are essentially uncompiled code. Thus when you want a group of macros want to collaborate, you do it by building up code fragments which are then parsed to be used by the next macro. That's going to hurt performance, error messages produced, etc. without heroic work by the compiler.

Maude's I/O system has no limitations relating to "reflective towers". The best independent review of Maude I've read is http://www.lshift.net/blog/2006/06/05/language-design-in-maude and http://www.lshift.net/blog/2006/06/11/maude-io

Problems with reflection begin when there is communication between modules or components, and use of reflection on that communication.

A recursive procedure is not a recursive process, and so it is with reflection.

I just finished coding an app that edits instances of itself. In a clever way it is doing computational reflection, w/o realizing it. No towers, though! Just one level was needed, plus a couple of extra objects that live in a different sphere of control from the app itself - so-called meta-level architecture.

Maude's I/O system has no limitations relating to "reflective towers".

The basic unit of I/O in a Term Rewrite System should be a module with an interface defining a set of types and rewrite rules, much like the basic unit of I/O in an OO language is an object reference and the basic unit of I/O in a functional language is a function.

Maude I/O is crippled. Until attempts are made to rectify that, it's difficult to make many assertions about why it remains crippled. I suspect, however, that supporting reflection on Maude modules passed as input and output between other modules would suffer the various problems I describe above.

A valid question might be: how much will compatibility with reflective towers hurt support for I/O based composition, if I/O were first-class in Maude?

how much will compatibility with reflective towers hurt support for I/O based composition, if I/O were first-class in Maude?

There are many ways to do I/O based composition, so I'm not sure what you are looking for.

The basic idea of an actor in the IOP looking like just another term is "first-class". - So I don't see your question as difficult.

Are you talking about performance, or what? Make Z-Bo smart. ;-)

Not that it matters any more, but at the time, what I wanted to do with Maude was wrap MIT's Alloy Analyzer w/ IOP and then use Maude as a metalanguage for Alloy. I can't remember exactly what I was hoping to accomplish, though - probaaaaably something on the level of using Alloy within Maude.

[Footnote: I was basing my coding off of Clavel's paper "Strategies and user interfaces in Maude at Work", and glancing at the ITP/OCL tool source code.]

The basic idea of an actor in the IOP looking like just another term is "first-class". - So I don't see your question as difficult.

"First class", to me, means that the feature is orthogonal, and that all other properties of the language apply. I doubt this can be said of IOP actors as viewed from Maude. For example, you cannot do Maude reflection on those IOP actors.

I wouldn't consider an 'actor' to be the ideal I/O unit for Maude. Maude is a term rewrite system. If a functional programming especially features the ability to communicate functions, and OO especially features the ability to communicate object references, then I/O in Maude should properly be the ability to communicate term rewrite rules (or rule-sets). Anything less is second-class.

Socket I/O is second class. IOP is second-class - it isn't even part of Maude, really.

You've been saying that Maude I/O is not hindered by reflection. I'm suggesting that Maude I/O is so neglected and decrepit that you cannot see its upper limits, or how the glass ceiling involving reflection would eventually hinder its growth to first-class Maude I/O in large projects.

What's the killer app for upper limits of I/O? CEP frameworks?

Distributed programming in general is at the 'upper limit'.

It is necessary to distribute code components to achieve performance (latency, bandwidth), resilience (regeneration after node loss), scalability and load balancing, disruption tolerance (i.e. for remote control, mission continuation), and more. Distributing data alone is insufficient.

It's worth noting that even web-browsers are all about shipping code around. High performance UI involving shared resources (such as shared documents) and distributed participants will, I suspect, be among the "killer applications".

I suspect distributed data-binding will also become a killer app. I.e. the ability to perform a "real-time search" that keeps you up-to-date continuously on the result of a query that joins multiple systems and databases. There's a good chance this sort of distributed data-binding will become a basis for UIs. It'd be highly resilient, disruption tolerant, support caching automatically, allow for fallback sources of certain data, and be demand-driven (no cost if no subscriber). Distributed data-binding benefits greatly from distribution and sharing of common functions (i.e. for filtering, combination, fallback) on a multi-cast network; doing so can reduce total bandwidth, storage, and processing costs by up to an exponential factor (O(N) -> O(log(N)) for N users).

Complex event processing (CEP) would be included under the umbrella of distributed programming. CEP itself has some theoretical problems to work out, though. I.e. multiple observers of the same events can trigger different result events based on when they started observing, latency to different sources, etc. It also doesn't scale all that well due to the internal state that must be maintained. I can't advocate 'full' CEP.

But weaker forms of CEP, such as distributed data-binding + simple event-processing (i.e. filtering, transforms) based on that data, will probably suffice for 95% of any CEP network and will have nicer properties for sharing computations, for eventual consistency, for distributed transaction semantics, etc. I suspect this 'simple event processing' framework will serve as a greater 'killer app' than CEP.

Correlating two different events as being close together in a nebulous concept of "time" (which is relative to the observer) could still happen at the edges of an event-processing network.

Those are the examples I was looking for you to flesh out. As anyone who has gathered requirements knows: Sometimes you just have to ask a question 10 different ways to get an answer. ;-) Now I basically just need to flesh out this and reflection.

There's a good chance this sort of distributed data-binding will become a basis for UIs. It'd be highly resilient, disruption tolerant, support caching automatically, allow for fallback sources of certain data, and be demand-driven (no cost if no subscriber).

Just curious, what researchers do you know of working on this? The closest I can think of is John Ousterhout' Fiz Framework and Jonathan Edwards Coherence Language -- neither of which are released to the world as binaries or source code.

I've encountered this in lesser forms (i.e. pushing filters, joining data... i.e. via DSL or API) from a few companies - Quantum 3D, Real-Time Innovations. Data-binding has a nice property of being asynchronous with display but still achieving eventual consistency, so it's appealing to these groups that wish to advertise real-time systems.

I've not seen much work at supporting it at the language level. I'm doing some work there, but I don't make a habit of pounding out papers.

Hi Leo,

Neither caller nor callee are keywords. They are both property names. There are actually four such properties typically accessible in JavaScript implementations:

1. (function-instance).caller // which function is the closest caller of this function on the call stack at this moment?
2. (function-instance).arguments // what is the current parameter variable array of the most recent call to this function on the call stack at this moment?
3. arguments.caller // Which function called the current function?
4. arguments.callee // What is the current function?

In ES3 and ES5, only the last is standard though all are typically present. Because these break encapsulation and prevent many optimizations, ES5/Strict prohibits all of them. Which is quite a trick -- to prohibit in strict code constructs that are not standard in non-strict mode. We agonized over how to do this. The hack we finally adopted was to mandate that strict functions provide all these properties with the behavior that they throw on any attempt to access them. This helps catch erroneous uses when trying to port non-strict code to strict.

The case we almost missed: non-strict (function or arguments).caller revealing a strict caller. Happily, we did catch and fix this it time. On non-strict functions, these need not exist, but if they do exist they may not reveal a strict caller.

As a result, the encapsulation of strict functions should really be safe even against non-strict code.