Does anyone have any reference material or thoughts on the idea of a method in an object oriented language always being allowed access to the sender of that message? It seems to be universally true that the sender of a message is unknown to the method unless explicitly passed as a parameter. I'm curious as to why that seems to be the default assumption. My gut instinct is that the rationale is connected with the idea of preserving encapsulation, but I wonder if that's really been tested or if that's more of a gut feeling that's been passed down through the years? (I know of no language that would prevent a sender from sending a reference to itself along as a message parameter - and indeed that is a common way around this limitation when the need arises. Why enforce the barrier in one case in the name of encapsulation, but not another if that is indeed the reason?)

## Comment viewing options

### Coupling and Delegation

If the callee is to know the caller in any useful capacity, it must know something about the caller... such as the caller's type or an interface. Thus, this introduces a coupling between the callee to all possible callers.

This in turn makes delegation of tasks difficult. I.e. if a caller wishes to wrap up a behavior in a Functor object that calls a method, that won't be easy to do. One would need to add the caller's special interface to the functor object.

Basically, the idea seems like a whole lot of pain with very little gain. What is it you believe you'd achieve by having callee know caller?

### Reasons for and against

Reasons for such a thing.

* Debugging; many systems allow a function to inspect the call stack, which does provide information (often with types erased) about calling objects and contexts.
* As a crude means of implementing things like dynamic scoping (in a dynamically typed context) or access control (if (caller != approvedCaller) throw AccessException; and stuff like that).
* Some aynchronous messaging protocols provide this by default, mainly to ease construction of a return message.
* Because it's available. :)

Note that I'm not endorsing either; but they are there.

Reasons against:

* As noted above, it increases coupling between caller and callee.
* Security issues--if a caller is to invoked untrusted code, it might want some guarantee that the callee won't inspect (or modify!) its state inappropriately.
* There are generally better ways to do whatever it is you're trying to do--the caller can always be passed in explicitly. Some of the "better ways" require explicit language support (visibility restrictions in OO languages, explicit dynamic scoping, etc)

### Reflection

I'd figure that some sort of reflection would be the obvious way to get information about the call stack (ignoring the problems presented by TCO).

Anyhow, aren't many of the listed pros and cons applicable to reflection in general?

### Reflection is a design bug

Indeed, those pros and cons apply to reflection in general. And note just how weak those 'pros' really are! weak privacy properties, a wide boundary for covert channels, poor support for a flawed security model, an inefficient memory model for deep and modular functional and procedural composition, and unnecessary coupling including (in the above case) a binding of messages to a "call stack" which harms concurrency. What's not to love?

Reflection is one of those design-bugs that people like to call 'features'... it's like a little baby dragon, all soft and cute and lovable while playing with toy programs. Then your programs grow larger, distributed, extensively modular, and it bites your arms off.

Dynamic scoping is a useful property. Make some form of it explicit. Allow the whole dynamic scope to be saved and restored, such that thunks can be associated with the dynamic scope in which they were created rather than in which they were executed, and to allow scope barriers be introduced (e.g. between the caller and a script interpreter). Make dynamic scope 'immutable' such that the continuation changes its scope rather than modifies the scope others are using, as this is good for concurrency and distribution. Put the 'System' information in dynamic scope, i.e. for logging errors and reporting to 'the console', thereby allowing system information to be changed. If you must use something like ACLs, then have a sender earlier in the call chain inject a challenge-reply object into the dynamic scope (which can respond to a challenge with a signature).

But avoid reflection! or at least limit its scope. For scalability, for security, for privacy: the callee shouldn't be able to "reflect" on the caller, and vice versa - the caller shouldn't be able to see the implementation-details of the callee, not even for resumable exceptions. (Put resumption and other error-handling policies in dynamic scope, too, if necessary.)

### Mirrors

If you haven't already, you should read the paper referenced in this LtU post and the paper passingly mentioned on LtU here. Probably the latter first, then the former. But yes, I certainly agree that reflection shouldn't be something anyone can do any time they feel like.

### Protection unless explicitly relaxed?

I know of no language that would prevent a sender from sending a reference to itself along as a message parameter - and indeed that is a common way around this limitation when the need arises.

This seems already to provide the argument against—it would be nice (as Scott Johnson mentions above) for a caller to know that its callee can't ‘see’ it unless it explicitly reveals itself. This seems much like the difference between passing a variable by reference to a C function, in which case all sorts of mutations and trickery are only to be expected, versus calling a function that unexpectedly modifies global state.

### But it wasn't me, I swear

The explicit argument approach has the benefit of enabling delegation. The immediate most calling code might be part of some code generator boilerplate code, or a proxy of some sort.

What it sounds like you want is an answer to "whose message is this?" A common answer is to have authors sign their posts. Otherwise you might blame the messenger.

### It seems you are arguing

(Ack, meant to be threaded as RE: "Reflection is a design bug" by dmbarbour)

It seems you are arguing against a specific form of reflection, rather than reflection as an outright concept.

Something that can look at itself is NOT a design bug. I don't understand how you tie reflection as a general concept to large, distributed, modular systems. I've never seen this argument in the literature, anywhere. Moreover, I see counterexamples. For example, the Maude Programming Language is a reflective term rewriting system with support for equational programming and deterministic and nondetermistic problem solving. It is distributed and high-performance, although I still believe much of it is still single-threaded (likely ensuring it has much more potential for performance improvements on multi-core machines everyone is in a kerfuffle over). Most of the design flaws in Maude have nothing to do with reflection whatsoever (e.g., Maude I/O) as far as I can see. I'd welcome enlightening comments that say otherwise, of course, and reserve the right to change my mind upon hearing a convincing argument.

There are other interesting reflective systems in the literature, too. For example, REFAL by Vladimir Turchin, although this approach depended on his Turchin Supercompiler model of programming. Actually, supercompilation brings up a good point, because one of Turchin's principles of metasystem transition was that the system ought to be simple enough to analyze itself. Thus, reflective systems should be remarkably simple and easy to understand.

Ad-hoc reflection, however, is most definitely questionable design. Just as other ad-hoc design techniques like ad-hoc polymorphism tends to be misused.

Reflection in CS literature is generally poorly discussed, and, as Basile STARYNKEVITCH recently quipped on LtU, usually forgotten by academics.

### Something that can look at

Something that can look at itself is NOT a design bug.

Even objects, as a default, carrying a pointer to 'self' is problematic for partial-evaluation and object-graph rewriting optimizations. The ability for a program to look at itself ultimately ties the implementation and memory structure of the program to the source structure.

I don't understand how you tie reflection as a general concept to large, distributed, modular systems.

I agree that some limited-scope forms of reflection are unlikely to be problematic. It's really the ability for a module to have a non-local impact or make non-local observations - including non-parametric observations on inputs to the module from outside modules (i.e. asking for the class of an input given by interface) - that spells trouble for security, distribution, and scalability.

Problem is, I see reflection as nearly being defined by the violation of parametricity. "Reflection" is the ability to take an arbitrary program unit (even one that you didn't create) and reflect upon it - see how it ticks under the hood, or what it's made of. Use of reflection is to make decisions based on this observation, which introduces the problematic coupling between one component and the implementation of whatever other component created the object you're observing.

The mere ability to represent (i.e. quote) code as data, and later interpret it (in a possibly modified language, or via a macro system) is not, by itself, reflection. Reflection is a form of meta-programming, but not all forms of meta-programming involve reflection.

Most of the design flaws in Maude have nothing to do with reflection whatsoever (e.g., Maude I/O) as far as I can see.

I haven't used Maude, and my only experience with a similar language (CafeOBJ) was a semester in university on automated theorem proving. I, therefore, can't speak on it overmuch. But I can suggest that the flaws of reflection could possibly be masked by the flaws with I/O, or perhaps reflection is challenge that makes inter-module I/O more complex and has thus delayed effective integration.

Problems with reflection begin when there is communication between modules or components, and use of reflection on that communication.

supercompilation brings up a good point, because one of Turchin's principles of metasystem transition was that the system ought to be simple enough to analyze itself. Thus, reflective systems should be remarkably simple and easy to understand.

Sure, a system should be as simple as possible (and no simpler). But your conclusion doesn't follow. The ability for a system to analyze itself doesn't imply the ability to analyze itself from within itself. That just introduces a new and interesting problems of non-termination and such (ask and reflect on: "will the answer to this question be no?").

Reflection is part of any system with reflection. When a system with reflection analyzes itself, it would also analyze the analysis of itself, and the analysis of that analysis of itself, and so on.

### Reflection is poorly defined

Reflection is a poorly defined term, or perhaps more accurately, solutions requiring reflection tend to indicate a poorly specified problem. It sometimes indicates a type of abstraction that does not yet have a typed equivalent. That set is much smaller than it once was, with well known approaches for overloading/ad-hoc polymorphism, polytypism, extensible records and variants, first-class labels, open data types and open functions, and polyvariadic functions, there's not much advantage left for real reflection IMO.

Reflection is part of any system with reflection. When a system with reflection analyzes itself, it would also analyze the analysis of itself, and the analysis of that analysis of itself, and so on.

When reflection research on Lisp systems was hot in the 80s, this was called the "reflective tower" IIRC.

### Poorly defined

Reflection is a poorly defined term

Right down there with 'meta-programming', eh? You won't hear an argument from me claiming it to be well defined. I can only say that my own understanding of 'reflection' as distinct from other 'meta-programming' techniques is that reflection happens to involve observing (and potentially modifying) the runtime structure of a program in a manner that rapaciously violates encapsulation.

I place reflection near meta-object protocol on the list of damning 'features' that harm the mature-language upper limit for optimization, scalability, security, flexibility, safety, and other nice properties. Every language has inherent limits to how effective it will be in full maturity, based on decisions made early in design (not just for the core language; common libraries matter, too). Reflection and meta-object protocol each do much to lower that glass ceiling.

I'm much more fond of macros or extensible grammars, partial evaluation, scatter-gather code styles (like rules-based and logic programming), and staging as meta-programming techniques.

solutions requiring reflection tend to indicate a poorly specified problem. It sometimes indicates a type of abstraction that does not yet have a typed equivalent.

Poorly specified problems don't require reflection, or much anything other than better specification. The use of reflection in those circumstances probably a consequence of exploratory programming. If the language didn't have reflection, but instead had some other feature unnecessary to the solution or problem, you'd probably see that other feature being used instead.

The latter is more common - the use of meta-programming techniques available to a language in order to work around language limitations or weaknesses. That's one of the big reasons for meta-programming, after all... providing the tools to Greenspun whatever you need.

But when meta-programming is used on the grand (non-local) scale, I believe it suggests a deficiency that must be fixed in future generations of the core language...

Something still missing from many languages is the ability to gather code and data scattered across many source files and combine it in an ad-hoc manner without side-effects. (I.e. you can do it in Lisp or Scheme with side-effects and mutable state, but that introduces ordering issues and such should any two constructed objects depend on one another.) Support for this in the language would probably avoid huge classes of code-walker meta-programs and preprocessors.

The sort of "well-known approaches" you list would also qualify, but it's worth noting that several of them (open data types & open functions, along with various classes of polymorphism, for example) are simply limited forms of the above 'provide, gather, combine' pattern. Too many techniques tackling the same problem demands a generalization of the technique.

this was called the "reflective tower" IIRC

You probably recall correctly. I saw the term "reflective tower" in Chapters 1 of the Maude 2 manual, plus the document Reflection in Rewriting Logic and its Applications in the Maude Language (1997), looked up when reviewing John's comments.

I suspect the excitement about these 'reflective towers' was about as relevant to productivity and utility of languages as the excitement over discovering that a Turing Complete language can emulate any other...

### macros over meta-object protocol?

I'm guessing you have some kind of highly dynamic, syntax changed out from under me view of meta-object protocol? Because in my view a good meta-object protocol is a typed, sane version of macros.

### Good meta-object protocol?

I'm guessing you have some kind of highly dynamic, syntax changed out from under me view of meta-object protocol?

Maybe so. I've seen meta-object systems in Ruby, Smalltalk, Lisp, and Python, each of which are decidedly 'dynamic' languages.

I also accept as axiom that modules and other forms of code-you-don't-control will be needed in any non-toy project, such that it's politically impossible to discipline all code used by a project. This tends to have me thinking that the ability for programmers to achieve predictable properties under deep composition of grey-box code is among the top features for a programming language.

a good meta-object protocol is a typed, sane version of macros

And where can I find this alleged "good" meta-object protocol?

Meta-object protocol tends to apply globally, i.e. modifying all instances of an object of a given class, or even all classes. It is difficult to ensure these modifications 'compose'. Modules using the meta-object protocol can easily conflict, or break modules that weren't using it. Worse, sometimes they do compose but it depends on hidden details such as the loading order for the modules, so you might get two modules working correctly but then they break due to some tweak in seemingly unrelated code. And that's even before considering security for mobile code and such.

Macros apply locally, to a region of source-code. So long as you avoid code-walking macros (which are problematic), macros are modular. Hygienic macros can also be defined, of course, but even non-hygienic macros are safe (supposing the environment in which the macros are defined is safe). Macros may or may not be typed; it doesn't really matter if they are typed, so long as the IDE can report errors in terms of the macro prior to expansion.

I'm curious as to in which capacity you believe a meta-object protocol is a "typed, sane version of macros".

### Don't throw the meta-object protocol out with the bathwater

And where can I find this alleged "good" meta-object protocol?

I don't know - I should have warned in advance that my feelings on this stem from my own work.

Meta-object protocol tends to apply globally, i.e. modifying all instances of an object of a given class, or even all classes.

Yes, this is what I meant by highly dynamic MOP - I agree this is a bad aspect of e.g. the CL MOP.

I'm curious as to in which capacity you believe a meta-object protocol is a "typed, sane version of macros".

Because the values computed by macros are essentially uncompiled code. Thus when you want a group of macros want to collaborate, you do it by building up code fragments which are then parsed to be used by the next macro. That's going to hurt performance, error messages produced, etc. without heroic work by the compiler.

### Maude's I/O system has no

Maude's I/O system has no limitations relating to "reflective towers". The best independent review of Maude I've read is http://www.lshift.net/blog/2006/06/05/language-design-in-maude and http://www.lshift.net/blog/2006/06/11/maude-io

Problems with reflection begin when there is communication between modules or components, and use of reflection on that communication.

A recursive procedure is not a recursive process, and so it is with reflection.

I just finished coding an app that edits instances of itself. In a clever way it is doing computational reflection, w/o realizing it. No towers, though! Just one level was needed, plus a couple of extra objects that live in a different sphere of control from the app itself - so-called meta-level architecture.

### Second class I/O

Maude's I/O system has no limitations relating to "reflective towers".

The basic unit of I/O in a Term Rewrite System should be a module with an interface defining a set of types and rewrite rules, much like the basic unit of I/O in an OO language is an object reference and the basic unit of I/O in a functional language is a function.

Maude I/O is crippled. Until attempts are made to rectify that, it's difficult to make many assertions about why it remains crippled. I suspect, however, that supporting reflection on Maude modules passed as input and output between other modules would suffer the various problems I describe above.

A valid question might be: how much will compatibility with reflective towers hurt support for I/O based composition, if I/O were first-class in Maude?

### how much will compatibility

how much will compatibility with reflective towers hurt support for I/O based composition, if I/O were first-class in Maude?

There are many ways to do I/O based composition, so I'm not sure what you are looking for.

The basic idea of an actor in the IOP looking like just another term is "first-class". - So I don't see your question as difficult.

Are you talking about performance, or what? Make Z-Bo smart. ;-)

Not that it matters any more, but at the time, what I wanted to do with Maude was wrap MIT's Alloy Analyzer w/ IOP and then use Maude as a metalanguage for Alloy. I can't remember exactly what I was hoping to accomplish, though - probaaaaably something on the level of using Alloy within Maude.

[Footnote: I was basing my coding off of Clavel's paper "Strategies and user interfaces in Maude at Work", and glancing at the ITP/OCL tool source code.]

### I/O based composition

The basic idea of an actor in the IOP looking like just another term is "first-class". - So I don't see your question as difficult.

"First class", to me, means that the feature is orthogonal, and that all other properties of the language apply. I doubt this can be said of IOP actors as viewed from Maude. For example, you cannot do Maude reflection on those IOP actors.

I wouldn't consider an 'actor' to be the ideal I/O unit for Maude. Maude is a term rewrite system. If a functional programming especially features the ability to communicate functions, and OO especially features the ability to communicate object references, then I/O in Maude should properly be the ability to communicate term rewrite rules (or rule-sets). Anything less is second-class.

Socket I/O is second class. IOP is second-class - it isn't even part of Maude, really.

You've been saying that Maude I/O is not hindered by reflection. I'm suggesting that Maude I/O is so neglected and decrepit that you cannot see its upper limits, or how the glass ceiling involving reflection would eventually hinder its growth to first-class Maude I/O in large projects.

### So...

What's the killer app for upper limits of I/O? CEP frameworks?

### Distributed programming

Distributed programming in general is at the 'upper limit'.

It is necessary to distribute code components to achieve performance (latency, bandwidth), resilience (regeneration after node loss), scalability and load balancing, disruption tolerance (i.e. for remote control, mission continuation), and more. Distributing data alone is insufficient.

It's worth noting that even web-browsers are all about shipping code around. High performance UI involving shared resources (such as shared documents) and distributed participants will, I suspect, be among the "killer applications".

I suspect distributed data-binding will also become a killer app. I.e. the ability to perform a "real-time search" that keeps you up-to-date continuously on the result of a query that joins multiple systems and databases. There's a good chance this sort of distributed data-binding will become a basis for UIs. It'd be highly resilient, disruption tolerant, support caching automatically, allow for fallback sources of certain data, and be demand-driven (no cost if no subscriber). Distributed data-binding benefits greatly from distribution and sharing of common functions (i.e. for filtering, combination, fallback) on a multi-cast network; doing so can reduce total bandwidth, storage, and processing costs by up to an exponential factor (O(N) -> O(log(N)) for N users).

Complex event processing (CEP) would be included under the umbrella of distributed programming. CEP itself has some theoretical problems to work out, though. I.e. multiple observers of the same events can trigger different result events based on when they started observing, latency to different sources, etc. It also doesn't scale all that well due to the internal state that must be maintained. I can't advocate 'full' CEP.

But weaker forms of CEP, such as distributed data-binding + simple event-processing (i.e. filtering, transforms) based on that data, will probably suffice for 95% of any CEP network and will have nicer properties for sharing computations, for eventual consistency, for distributed transaction semantics, etc. I suspect this 'simple event processing' framework will serve as a greater 'killer app' than CEP.

Correlating two different events as being close together in a nebulous concept of "time" (which is relative to the observer) could still happen at the edges of an event-processing network.

### Thanks

Those are the examples I was looking for you to flesh out. As anyone who has gathered requirements knows: Sometimes you just have to ask a question 10 different ways to get an answer. ;-) Now I basically just need to flesh out this and reflection.

There's a good chance this sort of distributed data-binding will become a basis for UIs. It'd be highly resilient, disruption tolerant, support caching automatically, allow for fallback sources of certain data, and be demand-driven (no cost if no subscriber).

Just curious, what researchers do you know of working on this? The closest I can think of is John Ousterhout' Fiz Framework and Jonathan Edwards Coherence Language -- neither of which are released to the world as binaries or source code.

### I've encountered this in

I've encountered this in lesser forms (i.e. pushing filters, joining data... i.e. via DSL or API) from a few companies - Quantum 3D, Real-Time Innovations. Data-binding has a nice property of being asynchronous with display but still achieving eventual consistency, so it's appealing to these groups that wish to advertise real-time systems.

I've not seen much work at supporting it at the language level. I'm doing some work there, but I don't make a habit of pounding out papers.

### Bottom line

If the receiver must know the sender, then you've created a contextual relationship between the receiver and the sender. Thus you've actually inserted an additional but hidden object into your collaboration model, but have not as yet assigned any structural responsibility to this object.

If you really need to know, model it. Introduce a new object, model the collaboration. This forces you to do structural analysis of the problem domain. It also forces you to maintain simple Moore state machine models for object state machines. There are many ways you can model such knowledge, too. For example, you can completely decouple the traditional GoF Observer pattern by sticking in a Registry object as a coordination point for publishers and subscribers. The important thing to note here is that you are capturing information structurally.

If you do not model it explicitly, then you will proceed down a slippery slope known as Presentation-Abstraction-Control, or, more generally, Frame-Oriented Programming: Every part of your program gets transformed from a tree into a lattice, thus creating spaghetti code everywhere. You suddenly become dependent on implementations, everywhere. You suddenly have to know the type of every object in your system. Type inference is no good at that point. Neither is polymorphism.

This is especially important if you are modeling a system where via metalevel architecture you can substitute message passing mechanisms.

However, more basically, the more structural information you can encode as structure, the more meaningful problem domain classes you can identify, the easier it will be for you to compose and simulate dynamic systems.

For more details, see Executable UML by Balcer and Mellor. Notably, Chapter 14 on Mongo Controllers: "The first rule of control partitioning is you do not create controller objects. ... The second rule of control partioning is you do not create controller objects." (paraphrased from memory).

### In javascript, up to

In javascript, up to recursion, you can access the caller and callee (both keywords/fields), in other languages you might get something similar for free from exception handling mechanisms.

Unfortunately, this breaks encapsulation -- wrt security sensitive properties like integrity and confidentiality -- by default. Without other support, it's hard to code defensively. Eg, you're forced to trampoline your own code.

### EcmaScript 5 Strict fixes this

Hi Leo,

Neither caller nor callee are keywords. They are both property names. There are actually four such properties typically accessible in JavaScript implementations:

1. (function-instance).caller // which function is the closest caller of this function on the call stack at this moment?
2. (function-instance).arguments // what is the current parameter variable array of the most recent call to this function on the call stack at this moment?
3. arguments.caller // Which function called the current function?
4. arguments.callee // What is the current function?

In ES3 and ES5, only the last is standard though all are typically present. Because these break encapsulation and prevent many optimizations, ES5/Strict prohibits all of them. Which is quite a trick -- to prohibit in strict code constructs that are not standard in non-strict mode. We agonized over how to do this. The hack we finally adopted was to mandate that strict functions provide all these properties with the behavior that they throw on any attempt to access them. This helps catch erroneous uses when trying to port non-strict code to strict.

The case we almost missed: non-strict (function or arguments).caller revealing a strict caller. Happily, we did catch and fix this it time. On non-strict functions, these need not exist, but if they do exist they may not reveal a strict caller.

As a result, the encapsulation of strict functions should really be safe even against non-strict code.

### Io

Io does this. It is, in fact, used rather heavily. If this is a good thing is a matter for debate.