How does Lisp do that?!?

Morning everyone. I have a question about the internals of Lisp (as a family) and I came here hoping someone may be able to answer my question.

I just recently watched the Rainer videos again, and I was completely amazing at the reflective, or maybe the proper word is introspective capabilities of Lisp, espicially while a program was running. So I finally decided that I wanted to know how this was possible, and why it was implemented in more languages to such a scale. What portions of Lisp contribute to this? Is there one specific feature, or is it more like a group of features?

Thank you everyone.


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Try it and see

The bad news is that you will have to learn to program in Lisp to understand this. The good news is that this is a fun thing to do.


I'm not sure learning to program in Lisp is really required to appreciate what facets of its design allow the high level of run-time introspection and support for incremental development. That said, I think anyone who has the slightest interest in programming languages should be forced to learn at least a Lisp dialect or two (in addition to a lot of other languages!).

Most of the run-time introspective/mutative abilities aren't specific to Lisps anymore. For instance, Python's namespace model [1] actually allows some stuff trivially that, AFAIR, wouldn't be doable in Common Lisp (well, it would be doable in CL, but would require recompiling a much larger chunk of the program). This is no slight against CL; the restrictions on CL's top-level environment are intended to make efficient compilation feasible, so that accessing a top-level-bound variable does not require a name look-up. I think the reason you don't see Python programmers (or programmers in other popular, dynamic languages) develop their programs in the incremental style popularized by Lisp programmers (where you have a long-running version of your program that you continually refine by adding new function definitions and updating existing ones) is more related to a difference in mind-set rather than being due to any technical limitations.

Anyway, if you have any specific questions, I'm sure there are a lot of people who'd be willing and competent enough to answer them.

[1] All namespaces are reified as dictionaries. Function namespaces (i.e. function-level environments) are, however, read-only at run-time to allow for efficient bytecode compilation (local variable references are usually translated to numerical offsets used by the LOAD_FAST/STORE_FAST instructions). Every other kind of namespace is fully run-time mutable.

from the standpoint of a switcher between Python and CL

I develop in both CL and Python (nearly exclusively those two these days), and one major advantage to the typical Python dev-style (edit, restart, repeat) is that it removes a whole class of possible errors, and leaves the programmer with less to worry about regarding the state of the program, so he (I) can concentrate on the new or bugfixed code.

In contrast, editing functions in a running Lisp is fraught with anxiety for me, as I'm wondering if there's some difference I haven't thought about that will result in corruption of the running system.

Edit. Kill. Restart.

I tend to occasionally kill my Lisp process and reload my code.

I keep wondering if I'm the only one doing this, or somehow doing it wrong, but hey. ;) Maybe a lighter touch would just be to delete packages or something.


I do this in Common Lisp too but seldom in Emacs Lisp or Erlang where state is more neatly partitioned. Marco's SLIME Inspector has helped me rescue a few Common Lisp cores though.

I do. Mostly because I have,

I do. Mostly because I have, in the past, managed to code myself into a corner where the code-on-disk required a (much) previous instance of the code-on-disk as a runtime environment and thus didn't work all too well in a fresh image. So once in a while, I double-check taht what I have saved is what should be saved, then exit the running lisp, start a fresh image and load everything up again. However, it's only happened once or twice since and I realised I was in nasty dependency hell as I was about to check for it.

To Answer Your Question...

By Lisp's "introspective capabilities," I assume you mean its ability to inspect and modify its source code at runtime.

The way this is done is simple: It doesn't add features, so much as it removes features. Rather than "defining" a function, you assign a functional object to a variable. Naturally, this means you can inspect the contents of the variable, and assign a new value to it.

Thus, (defun f (x) (y x z)) is (about) equivalent to (setf (fdefinition f) (lambda (x) (y x z)). (It's (fdefinition f) instead of simply f because of some details of Common Lisp's semantics.)

(None of that will have made sense if you're not familiar with first-class functions. If not, I highly recommed you learn Lisp, as you could clearly get a lot out of it.)


Introspective capabilities of Lisp are an instance of what's often called reflection or "reflective programming",
and this wikipedia page touches on some examples:

Basically, nothing stops you from making some object (ie that's reachable dyanmically at runtime) for every important concept involved in the programming enviroment. You can make first class objects to represent everything, if you don't mind the tedious work, and if conventions in the language don't fight back. You can do it in every language, but the lower level they are, the less friendly they are toward reflection.

What gets complex is when you want to be able to change these objects when the system is running, and have these changes affect the current system. In low level languages like C and C++, it would require all the code to be driven by the dynamic metaprogramming system, instead of the normal static class hierarchy used in (say) C++, and this goes entirely against the philosophy and practice of libraries in C++. But in principle, C++ libraries can be written that are as fully reflective as Lisp and Smalltalk. But C++ zealots might hate it on principle, though. (Otherwise, it would already have gained favor.)