Lambda the Ultimate

The bad news is that you will have to learn to program in Lisp to understand this. The good news is that this is a fun thing to do.

I'm not sure learning to program in Lisp is really required to appreciate what facets of its design allow the high level of run-time introspection and support for incremental development. That said, I think anyone who has the slightest interest in programming languages should be forced to learn at least a Lisp dialect or two (in addition to a lot of other languages!).

Most of the run-time introspective/mutative abilities aren't specific to Lisps anymore. For instance, Python's namespace model [1] actually allows some stuff trivially that, AFAIR, wouldn't be doable in Common Lisp (well, it would be doable in CL, but would require recompiling a much larger chunk of the program). This is no slight against CL; the restrictions on CL's top-level environment are intended to make efficient compilation feasible, so that accessing a top-level-bound variable does not require a name look-up. I think the reason you don't see Python programmers (or programmers in other popular, dynamic languages) develop their programs in the incremental style popularized by Lisp programmers (where you have a long-running version of your program that you continually refine by adding new function definitions and updating existing ones) is more related to a difference in mind-set rather than being due to any technical limitations.

Anyway, if you have any specific questions, I'm sure there are a lot of people who'd be willing and competent enough to answer them.

[1] All namespaces are reified as dictionaries. Function namespaces (i.e. function-level environments) are, however, read-only at run-time to allow for efficient bytecode compilation (local variable references are usually translated to numerical offsets used by the LOAD_FAST/STORE_FAST instructions). Every other kind of namespace is fully run-time mutable.

By Lisp's "introspective capabilities," I assume you mean its ability to inspect and modify its source code at runtime.

The way this is done is simple: It doesn't add features, so much as it removes features. Rather than "defining" a function, you assign a functional object to a variable. Naturally, this means you can inspect the contents of the variable, and assign a new value to it.

Thus, (defun f (x) (y x z)) is (about) equivalent to (setf (fdefinition f) (lambda (x) (y x z)). (It's (fdefinition f) instead of simply f because of some details of Common Lisp's semantics.)

(None of that will have made sense if you're not familiar with first-class functions. If not, I highly recommed you learn Lisp, as you could clearly get a lot out of it.)

Introspective capabilities of Lisp are an instance of what's often called reflection or "reflective programming",
and this wikipedia page touches on some examples:

http://en.wikipedia.org/wiki/Reflection_%28computer_science%29

Basically, nothing stops you from making some object (ie that's reachable dyanmically at runtime) for every important concept involved in the programming enviroment. You can make first class objects to represent everything, if you don't mind the tedious work, and if conventions in the language don't fight back. You can do it in every language, but the lower level they are, the less friendly they are toward reflection.

What gets complex is when you want to be able to change these objects when the system is running, and have these changes affect the current system. In low level languages like C and C++, it would require all the code to be driven by the dynamic metaprogramming system, instead of the normal static class hierarchy used in (say) C++, and this goes entirely against the philosophy and practice of libraries in C++. But in principle, C++ libraries can be written that are as fully reflective as Lisp and Smalltalk. But C++ zealots might hate it on principle, though. (Otherwise, it would already have gained favor.)