Lambda the Ultimate

So far aspect oriented programming (AOP) has felt nebulous to me, without a clear or specific agenda, expressing vague intent instead. (Hang loose, avoid inheritance, mix and match something something.) If you noted a specific way it addresses or contributes to splitting code annotation into different places, so it can be optionally rebound in different contexts, I'd be happy to hear your observation.

When I saw an HN post on code weaving in Golang, which mentioned aspect oriented programming (aop), I decided to read up on aop again, and found folks usually look to some feature in aop when they want any result like one I described. (Below I may quote from Wikipedia's Aspect-oriented_programming and from c2's AspectOrientedProgramming.)

So the conversation I wanted to have is indeed inoculated by aspect oriented programming, to the point I would have to draw comparisons, even though I meant something a bit different, which I will refer to as romp for rule oriented meta programming. (Yes, the verb to romp has funny synonyms when applied to code.) The concept I have in mind for romp is something like injecting macros based on explicit rules. Why use another term? Because aop was patented by Gregor Kiczales in 2002, so it's suicidal for any product to define itself in terms of aop, so you practically build infringement into the definition. Fortunately, I have little use for the notions of advice, join point, and point cut defined by aop, partly because they are too specific.

The general idea of meta-programming is to generate code, via code that produces code. Macros are the old-school concept that does this, because expanding a macro generates the actual code to be compiled or interpreted. Fortunately, macros are not patented. The term originates in the phrase macro instruction, meaning a big or high level instruction, so the word macro doesn't mean anything more exotic than big in the sense of expanding into smaller concrete instructions. Basically a macro is source code level abstraction, rather than runtime execution abstraction -- or at least, that is the opposition involved.

However, to ordinarily get free use of macros everywhere in code, you would have to write all code entirely in terms of macros, adding an obfuscating level of indirection, basically everywhere, just so you could change macro definitions.

The idea I had in mind is pretty simple. You ask, what if all the code had been macros? It's not all macros, but what if it had been? Suppose we make a tool that injects source level abstraction by inserting macros where now absent, but expanding into what it is now, unless you redefine the macros to do something else, usually to inject side effects like logging or statistics. This c2 quote is relevant:

In a sense, AspectOrientedProgramming is the opposite of FunctionalProgramming. The core mindset of FunctionalProgramming is computation without relying on SideEffects; in a sense, the core mindset of aspects is adding SideEffects. Aspects are like the DecoratorPattern applied to functions.

If new behavior injected by macros was unobservable, you might as well not have done it. So if you can see any effect, it amounts to new dataflow as a side effect -- perhaps an evidence trail about what happened in the original code, for example.

Note another use of code rewrite is to narrow types, making an interface appear to be a facade doing less, just to provably constrain part of a code base to less than all the features in the full interface. (Ordinarily, the motivation for information hiding is to enforce locality, by permitting some things only in well-known places, so analysis can be local instead of global.) This sort of interface rewrite supports exploratory verification that some sub-systems do not use features, even though possible, because you can type-check as if absent. The synonym of 'to romp' we might use here is 'to frisk', meaning to pat-down the code to see whether it carries any use of api it should not (or you hope not).

Injecting behavior amounts to advice in aop, but with macros it is just expansion, and since you can see the new expanded version, it suffers less from control flow obfuscation. Where you can inject behavior amounts to a join point in aop, but in romp (rule oriented meta programming) everything is a join point if you can write a rule pattern or specific rule that matches, so everything you understand is join-able, and doesn't need a new named entity. (When it is true of everything, it's not a distinguishing characteristic.)

I'm inclined to use Lisp as meta-language when specifying rule declarations, but the target for injecting macros can be any language, including C (which is my focus currently). It isn't necessary to make a meta-language executable in an imperative sense, since you can simply generate it dynamically, and get the same effect as executing. So declarative semantics is sufficient. You could have as many meta-meta languages as you like, if they generate meta-language in time for consumption by the compiler applying rules to inject macros.

Details in my approach to applying this probably aren't interesting, so I'll stop here since the above covers the basic idea, which might be restated as follows. You can define macros somewhere else, as annotations, without lacing them into code you want changed. Injecting macros can be done by rule, either quite general ones (e.g. patterns) or very specific ones (first use of X in function Y). How the result of macro expansion gets seen or used depends on your development environment.

Thanks for taking time to post your ideas. (I'll try to reward posters with feedback if no one else does, albeit perhaps a bit slowly.) Is there a summary sentence or paragraph that gives shape to the approach you want to build up?

I see isolation of accidental stuff as partly a naming issue, because when you separate things that refer to a common entity, they need a way to agree on names for it. (Or at least, a compiler must be able to figure out the name in common.) I had in mind tree-shaped scopes, named by paths, which works okay with a virtual file system, if a compiler consumes source code making statements about code inside a vfs. Presumably associating separated things is a problem everyone must solve with annotation, if context is not merely adjacency.

I see programming languages being good for something specific, namely for stateful UIs that are deeply nestable and mostly self-contained.

I rarely work on UI, so seeing PL as having a UI focus is hard for me. But I'm used to a related idea that interfaces are like languages in general -- that the API of a system is a language for programming it. So whatever you are driving, the way it is done has a kind of language in terms of entities, operations, and state descriptions.

The word extension occurs a lot in your discussion. Architectures oriented toward extension seem to get a lot of coupling in runtime details, with many dependencies, instead of (as one might hope) high independence of parts. Perhaps this is because communication is often expressed in highly runtime specific ways (you will be in my language as well as synchronous, etc) instead of something more message oriented.

Thanks for taking the time for that reply! I felt myself being more long-winded than usual in that post.

I could try to give a paragraph to describe my approach at the highest level, but I wouldn't know where to stop. I keep shifting my attention to different parts of the problem: Things like syntax; dispute arbitration; knowledge representation; knowledge communication orchestration; programs as bags of constraints that merely suggest causality flows rather than spelling out every step; and recently, memory layout.

Regarding the particular issue of separating accidental details...

An important aspect of my high-level approach is that I think of every program as having a back door. Its interpreter, runtime libraries, hardware, etc. may be compromised, and there's nothing the program can do about it. If the program opens the back door and bravely faces the person in control, then we can simplify I/O and error-handling designs. We no longer need a front door for I/O, and the program can directly ask how to deal with errors.

Where I said "language runtime extension" before, I'm referring to the set of tools the user at the back door has on hand. If a programmer wants to separate incidental details from the main program, they can write a main program that frequently consults the back door, and they can separately offer a supplementary tool. If a user invokes the program directly, they may find themselves overloaded with consultations, but using the tool, they can set up auto-handlers for most of them.

It's possible the program and its supplementary tool will be rather coupled to each other, but at least the separation you're talking about is achieved. The original developers may engineer upgrades either for the tool or for the main program independently. Likewise, dedicated users may use reverse engineering (and reading documentation, etc.) to engineer their own compatible upgrades.

I was about to say a few more things about naming and coupling, but I'm afraid of overloading this comment with digressions.

I like long-winded, in the sense it takes a few words to say anything interesting, so non-trivial worthwhile notions cannot easily be short. (For folks with long attention spans, hundreds of words are a natural length needing no summary. But point prioritization helps convey intent to focus on one thing or another. Central questions tend to be short, for example. So summary is agenda in abstract, and helps separate foreground issues from background subtext -- action from setting.)

Nice vein of organizational ideas you're exploring, about models of infrastructure. We probably need more well-known examples of model variants, so options come to mind more quickly as folks tune design ideas. I often work with people locked in a death grip with just one model, so other options don't register, and sound like speaking-in-tongues when I depart from the expected cliché. Monolithic centralization is often assumed, as if air travel is impossible without getting permission from the flight control tower, so talking about drones will sound like gibberish to someone thinking airport.

The back door metaphor seems useful.(A front/back spatial metaphor is somewhat horizontal, with front closer and back farther away, situating POV of a speaker at the front where an interface is, with backend away from interface). Sounds similar to trusted computing base (TCB), as far as you emphasize relying on something over which you cannot control, and therefore must accept the risks inherent in dependency. Since someone always has permission to go modify the trusted base, and that someone might be you when debugging for example, it doesn't help to have code on top make too many limiting assumptions.

A base or bed spatial metaphor is somewhat vertical, emphasizing a stack of layers, as in a cake, with trusted parts in a bottom foundation. I like the word bed for TCB because it's just one syllable, hard to confuse with other words, and rarely appears in tech contexts. So it would work okay in coining terminology about layers. However, one up-down orientation is somewhat monolithic, making the bottom feel more sacrosanct than it is, since moving a foundation can wreck a building. When coding you often do it anyway, so the base is not unreachable. Perhaps a fold spatial metaphor works slightly better, as if software is exotic origami with convoluted arrangement. If the bed is visible in the api where you can insert plugins, you can replace the TCB with dependency injection.

Software is very refoldable, so moving trusted dependencies around is not hard. The bed or back door of a module should be pluggable, and only needs to be folded the right way so access is easier, as long as you are that someone with permission to yank the tablecloth from underneath all the dishes anyway. (Of course, you don't want privilege escalation to be possible.) Folding software fluidly this way is really awkward in some architectures, when the bottom is riveted and cemented in place, so change involves too much demolition.

I've been working on painfully complex software many years now, so it's hard to think of it as one program rather than many cooperating independent entities. A model of lots of interacting parts is more fluid. But there's too much fragility (in what I see now) caused by some parts being frozen (riveted and cemented) as native OS processes, while other parts are frozen as ad hoc callback FSMs composed of webs of messages, timers, closures, queues, and object-based state machines. It would be simpler under a uniform abstract process model, so refolding was afforded by uniformity, consistency, and abstraction.

To me a pattern is a rule that applies to more than one place, and need not involve regular expressions, for example, which some people associate with patterns. (I don't think I would ever use regex when injecting macros by rule.) I see you are not referring to regex either. But your patterns still seem a bit text focused.

Using the language grammar as the basis for a pattern grammar would allow the programmer to capture fragments (inlined strings) of lexemes (language constructs).

For injecting macros by pattern rule, I don't think I would use language grammar as part of pattern grammar. I would prefer that patterns in meta-language refer to semantic concepts in a parse tree. I'd also include ideas like "before everything" and "after everything", which don't reference a parse tree node per se.

The more surface-oriented patterns are, the more fragile they ought to be, the same way that a C pre-processor working mainly with source text is awkward, error prone, and (um) intention-erasing.

So basically: if you give the programmer a way to iterate over the program parse-tree and a pattern matching syntax for the host language then reflection becomes very expressive.

Regarding expense, injecting macros need only occur when original source changes, if your intention is to track versions instead of freezing to an earlier specific version. Whatever the cost, it's cheaper than doing it by hand. I wouldn't aim to regenerate everything for every build in any case. [Edit: yes, I read your expressive as expensive, making this and the next paragraph a non sequitur of sorts.]

So far, the idea of what I might write patterns for does not strike me as expensive to execute. It seems less expensive than fully understanding the source, because it involves only recognizing where something happens and/or uses of relative relationships. If you wanted to completely rewrite everything by rules rather than inject macros, that would get more expensive, sure. But that seems more like an interactive developer-driven process anyway, with human UI latencies too.

As the patterns become more complex an efficient method of performing the search is needed.

I agree optimization is usually relevant. :-) I just think of it as interactive and incremental, rather than a roadblock.

This was discussed a little in the Infer topic.

I wasn't that interested in the Facebook project, though I do find their infrastructure issues interesting. (Generally I don't respond to feelers from Facebook recruiters, because I don't use Facebook, and it seems weird to work on anything you don't care about. It's easier to work at places that have no real public application face anyway, so one need not pretend to care about user interfaces.)

I think that John has the right idea in his comment above, this also reminded me of aspect weaving.

Yes, injecting macros by rule amounts to code weaving. But it would only be aspect weaving if you carefully mimicked a specific aspect feature.

Unlike the gather side, expressing projection of changes back on to the code seems difficult.

Write a new copy of the code with macros injected (and optionally expanded).

The issue of controlling permission to alter code seems to be key: without any form of permissions it would be a high-level approach to self-modifying code.

Yes permission matters. I've been treating that as an application detail I expect folks not to find interesting. I would allow meta-language to express constraints, which could be contradictory and result in errors of course. And using a virtual file system, there would also be virtual user level permissions for coarse grained effects.

Configuration does seem like one of the use case reasons to version code interfaces and implementation.

Thanks, I read that and also searched, but reward was sparse, especially that fragment spec which reads like a reference, providing more examples than background explanation. It needs more walls of text. :-) I like more vertical space used by natural language prose than quoted code examples.

I was unable to see any relation to my post until searching Google for Beta language metaprogramming. And then I could see it was used as an aspect weaving touchstone by some older books. I appreciate the effort.

I should've provided some context and a better reference.... IIRC you can leave "slots" in parts of your program (basically certain kinds of syntax nodes in an AST). A slot has a name and a syntactic category. You can then construct a fully realized program by plugging in code fragments from other files. The slot name is used to match a fragment with a specific slot and the category is probably used for simple checking.