Lambda the Ultimate

Don't twist this as an FP vs. OOP argument. C++ sucks on its own level with non-modular type checking, you can totally have modular type checking in a OOP language.

As for the OP, abstractions invariably leak, but we just have to deal with that. There is definitely a cost to reusing a library, and the benefit you gain from reuse should outweigh that cost; if it doesn't, don't use the library.

This is basically it -- the cited problem is with type-checking C++ templates. Lacking "concepts" or a similar mechanism, template errors are identified at the deepest possible level rather than the shallowest possible level. Taking that and turning it into a criticism of "libraries" is definitely a bridge too far.

Words like abstraction and reuse and cost/benefit have long led me astray.

Units of reuse aren't made in one big-bang moment. They evolve. You write code for one use case. Then maybe you find it works in another, and another. The organic process of reuse requires a lengthy gestation period. But our language and our libraries obscure this fact.

We imagine our job as programmers is to create abstractions. It is not. It is to understand the code surrounding us, that we interface with and use to create new things. Our job is to *understand* abstractions that have come before and proved themselves and maybe, occasionally, if we're very good and a little lucky, to create an abstraction that others will find useful. The conventional way we view libraries obscures this fact. It makes it seem like creating a unit of reuse is just a matter of running gcc/ar/ranlib. And so we're awash in libraries with prematurely ossified interfaces.

I still don't know what you are getting at, your thoughts are a bit too abstract. I think it might be related to Joel Spoleky's "Abstractions Leak" or the "Duct Tape Programmer" posts.

I'm mostly just working things out for myself, and looking for feedback on the process. What I have so far -- and what this post tried to summarize -- was that the world would be a better place if we all know a little bit more about our dependencies.

I think this is consistent with Joel Spolsky's posts. Part of the reason jwz is such a good programmer (and also part of Feynman's prowess) is this willingness to bounce between the two sides of abstraction boundaries. To truly understand that all abstractions leak is to be willing to learn about their internals, and if everyone did this we might even end up building less leaky abstractions rather than shackling ourselves to their leakier initial drafts.

I didn't have any twist in mind, I merely saw two cases that seemed to share a common type of problem: A promise of abstraction that bogs you with implementation details.

It might be C++ specific when it comes to OOP, though I'd say the problems are there for Python as well except the error messages are a bit easier to decrypt. Apart from that I don't have experience of other OO languages.

If a part of your codebase never kicks up an issue, it doesn't matter whether you understand it or not :) I tried to allude to pragmatic considerations when I said start with lowest-maturity/highest-risk.

---

And oh, guilty as charged, I'm pretty low-productivity. Productivity is over-rated.

With hindsight, perhaps the names I've used are confusing. Libraries aren't bad if you learn how they work. But in today's world they mostly seem to discourage poking inside them. We're more concerned with some speculative future issue like, "If I make a change how will I merge it upstream? I'll be maintaining my hack forever." Don't worry about that for now, just go understand how it works.

Thanks for working with my incompetence and taking the trouble to figure out what I meant :)

I'd just add that maturity = #users * time.

I almost feel we need a different name for immature libraries (which are now the vast majority of all libraries). That allows users to set expectations about the level of churn in the interface, and frees up library writers to correct earlier missteps. I think the Go language did this well, and lots of other projects do think about gradually firming up interfaces. But many projects also stay on one side or other of that divide. My impression is that open-source C++/java projects have historically cared about compatibility too soon, and the rails eco-system cares about compatibility too little. 6 years in, fairly standard gems are still constantly in a state of flux.

Another side-effect of a new name for immature libraries: it would utterly invalidate the title of my article.

This is less important and more subjective, but I think we also need more attention to deprecation and deletion from mature libraries. Mature libraries tend to be great from a stability or performance perspective, but bad APIs do creep in. Libc and the java standard libraries and Common Lisp have lots of ugly warts that don't look like they'll ever be excised. (Is this just because they were prematurely ossified back in the day?)

My pet peeve: Lisp's multiple flavors of equality are still with us 50 years later, leading later languages like javascript astray. The superior approach IMO is to use structural equality by default, and offload the semantics of eq to an operator that returns a stable address for a value. In my toy language I call this operator addr. So I'd replace (eq a b) in Common Lisp with something like (equal (addr a) (addr b)).

So I'd replace (eq a b) in Common Lisp with something like (equal (addr a) (addr b)).

... and then people would just write

(defun (eq a b) (equal (addr a) (addr b)))

and mutter vaguely insulting questions about why you didn't provide it for them. And when enough of them had done so, people would say, "hey, look, there are dozens of definitions of this thing, it ought to be a standard item."

a) That isn't what actually happened. In reality eq came first, in an era more miserly of compute cycles.

b) It *could* happen for some features, but you don't know that for sure. Short-circuiting the gradually-clarifying need for something is a recipe for bloat.

c) If it *did* happen, I'd argue the system was working as designed. That's kind of my whole point: that things should prove themselves before we freeze them.

Most programmers don't actually use eq that often. Python and ruby and perl still don't have it. No vaguely insulting muttering has resulted. (Is it really a big enough inconvenience to cause even muttered insult?) Most of the time eq is just an optimization, usually a premature optimization, and even otherwise one that can mostly be delegated to the implementation.

I think languages should provide opinionated defaults and empower users to change them if necessary. Wart lets you override the default equality comparison to simulate eq, or extend it to do so just for specific types or in specific scenarios.

Python's "eq" is the "is" operator. Ruby's "eq" is, confusingly, named "equal?". (I'm a fan of Baker's "egal" predicate, myself.)

I was corrected elsewhere as well.

It's been a while since we debated, but if I recall correctly we aren't really disagreeing, just focusing on different parts of the state space. I'm totally fine with black box modularity for mature libraries as long as we pay attention to the process of gestation, and don't treat all libraries as equally mature.

I'd also prefer to leave a little escape hatch for even seemingly-mature libraries -- just in case we find a better API or something :) But that should be covered by the usual deprecation best practices.

Part of the problem is what I refer to, for want of a better symbol, as the tragedy of the commons. To some extent such tragedies are unavoidable in society. The trajectory of bureaucracies made such patterns hard to see -- you had to watch them closely over decades. Software is great because we can see dysfunction laid out over a period of months. Since the problems are easier to spot, I tend to be optimistic that we'll be able to do better with free software. You get a lot from free software in exchange for a certain amount of "doesn't validate completely against spec." I just think we don't squeeze out all the benefit we could because of these mental shackles, these imaginary constraints on the changes we can make.

Well there's something to be said for looking at both sides of an abstraction during development (if that were never necessary then we'd never write any functions in the first place).

But I think that maybe what you're saying here is that, when a software problem is found, we shouldn't have to pierce multiple layers of abstraction to find it, but rather the bug should manifest as a specification error (and if bugs are otherwise possible, it's a failure of the specification that should have excluded such a possibility).

If that's what you're getting at, I agree completely. Programs should be correct by construction.

If that were to become true then yes, my entire point might be moot. Can you point me to any papers on 'correct by construction'?

I'm not sure why you use the qualifier 'during development'. Do you mean during development of the abstraction? I'm pushing for 'always'. Or 'during any development I do'.

My post was deliberately agnostic of language or platform. But I suppose I don't know much about the life of say a Haskell programmer.

So much as a bad process. But another saying besides my credo, "Don't manage labor; eliminate it", is also "One fact, one place, one time". Things work best if a specification is the single point of change.

Consider manually writing DAOs, DTOs, manually ferrying the data to domain objects, manually encoding visitors, manually doing change tracking, etc. There's a lot of points of failure in such a process.

Suppose an underlying field changes, or the order of serialization changes. How do you permanently avoid such issues? Such synchronicity issues in code can be handled through macro-expressivity or a custom DSL equivalent to a macro, and many additional safe ways, although one would ideally subsume all others.

It's a theory of organizations, which can be automated by computers acting on behalf of humans.

Don't repeat yourself, then?

I've definitely had some success, in the working world, formalizing hairy problems enough that the repetitive error-prone parts are mechanically derived from a formal spec, as you say.

... is not imports or names. Import polymorphism is a real problem, as Gilad discussed. But imports are fine just so long as you remember they are only for interaction with people (to aid with binding resolution). Imports are fuzzy. Imports should be dev time.

Yes, this is something I learned when I did Jiazzi. That the external configuration of imports in units/modules/components wasn't really that great of a feature, that all the use cases for configuration connections were either contrived or I was abusing the system to do something else that wasn't really related to modules (open classes).

Specific use-cases can reveal meany problems. But they are a biased lens. Many systemic problems cannot be observed through use-cases, especially when those use-cases are influenced by or described in terms of integration with an existing system. Issues such as software reuse, reconfiguration, and securable composition are unlikely to be observed in use-cases for any particular project. And yet there is great value in addressing them.

Wonderful, but I'm commenting on today's world. If you simply make module import from Java-like languages externally configurable, you aren't doing much because imports are almost always made with specific implementations in mind. If we are talking about alternatives to type parameters, then ya, there is a use case we can address with external linking. But the two concepts are very different.

Even if we use imports "only for interaction with people", that's enough to lead developers straight to dependency hell.

An individual has the time and energy to directly maintain a number of relationships that is relatively small compared to the population as a whole. In a group of N people, there are N^2 potential communication relationships. That simple law is why we need hierarchies and bureaucracies. And it has a significant impact on modular organization of software.

Relationships with people are subject to variations over time and space, shifting in response to markets, resources, and requirements. If the number of relationships is small, we could perhaps maintain all these relationships concretely, by hand at dev-time. But, in general, we must specify policies and roles and manage relationships and "interactions with people" - including imports - automatically.

If you don't handle it in the PL, you'll end up hacking it (horribly) with tools like autoconf, cpp, and cmake.

David, if I understand correctly, you're saying that hierarchies arise to organize a certain kind of decision so that large numbers of people are in sync and can work together.

Another possible solution: the population converges on using a specific solution over time. It requires individuals to be more conservative in the dependencies they rely on, and to grow an immune system that can assess new dependencies and choose between competing ones that provide overlapping functionality. Individuals may also go out of sync with the herd for short or long periods.

I used darcs for a couple of years but now I'm on git. That required me to learn to migrate my repos, something that an organization might delegate to one specialized little team in the basement.

I glimpse an analogue to the CAP theorem here. The 'market solution' allows individuals to go out of sync for periods of time in hopes of increased flexibility, and the ability to manoeuvre more quickly in response to changes in the environment. Hierarchies try to keep everyone in sync all the time, 'protecting' individuals who haven't grown an immune system yet, in hopes that obviating the need for redundant immune system management will provide efficiencies at scale. Both seem plausible as responses to the quadratic explosion you describe. Which is better depends on how much manoeuvring the environment requires.

In science fiction terms, we're talking about Star Trek vs Cyberpunk worldviews.

I did not mean to suggest a preference for global hierarchy, or whatever it is you're imagining with the "Star Trek worldview".

I'm quite fond of system heterogeneity and data model independence. I see heterogeneity as natural and valuable for evolving systems, relationships, and markets. I make much effort to address and support integration of heterogeneous systems in PL designs. (And heterogeneity benefits greatly from effective automation of relationships and imports.)

Hierarchy is typically a local structure in a larger, heterogeneous system. E.g. a company may have a hierarchy, but there are many different companies and the relationships between them aren't really part of any hierarchy. In secure modular software systems, small hierarchies will tend to exist in the resource models and distribution of authority.

A point to consider: even to "assess new dependencies and choose between competing ones" can become a O(N^2) effort if the number of competing solutions is proportional to the number of people or companies. (The prevalent "Not Invented Here" philosophy makes this a not unreasonable assumption.) More importantly, often the choice of competing solutions is a property of a relationship, not well isolated to any group.

The Enterprise is just one starship among many :) (Ok, I think that analogy's outlived its usefulness.)

Empirically it doesn't seem to me that there's quadratic explosion in competing choices. Do you see that differently? It also seems intuitively reasonable: there's just so many different problems. If everyone in the world was working on todo list apps or erlang compilers I'm sure there'd be a lot more todo list apps or erlang compilers. But fortunately the world seems to need both, and a zillion other things besides.

I'm trying hard not to characterize either side as a strawman. Let me know if my biases are betraying me. Our world is a lot more driven by hierarchies than people realize. My sense is that after favoring large organizations for a couple of hundred years, the pendulum is swinging back a bit toward many more, smaller organizations. That shift has implications for software that I'm trying to work out.

There is a roughly linear increase in competing products with population. The quadratic explosion happens in the number of comparisons needed to choose the best one (re: the effort to "assess new dependencies and choose between competing ones"). Similarly, integration code tends to grow quadratic with heterogeneous choices.

That you don't see quadratic explosions directly is because there are already practices and mechanisms to mitigate them. Packaging of decisions (e.g. Ubuntu comes with Gedit, no need for each user to consider the many text editors). Compilers often use intermediate languages or representations (to support multiple front-end languages independent of multiple back-end optimizers).

Competition is further mitigated by the general fact that, in any competition, there are usually a few clear leaders to choose between. The rest are mostly ignored, yet are still important for the health of the system (new leaders must come from somewhere).

While there aren't many Erlang compilers, that's mostly because (relative to more popular languages) there aren't many Erlang users. :)

(Trying to return to the original topic..)

Ubuntu and compilers are responsible for taking some set of possible configurations and testing them to make sure they work. I am assuming that such testing will always be necessary. At any point the integrator will choose between a handful of choices -- and support some small subset of them -- for every component of the system he's providing. This doesn't seem like a problem to me.

This my core disagreement (or difference in emphasis), I think: PL techniques will never suffice for avoiding DLL hell. You'll always need to be mindful of your (transitive) dependencies. That's a social issue, an education issue.

PL techniques will never suffice for avoiding DLL hell. You'll always need to be mindful of your (transitive) dependencies. That's a social issue, an education issue.

Hmm. I don't make a strong distinction. Cf. my article social aspects of PL design. We're really in the business of programming experience design (PXD) - a term coined by Sean McDirmid in a discussion with Jonathon Edwards in An IDE is not enough.

A wise PL designer will consider various social acts that contribute to development, dependencies, deployment, configuration, integration, maintenance, extension. Even if the vision is mostly of a solitary programmer, addressing rare social elements can improve the programming experience. A PL designer with a great interest in social experience or CSCW might pursue concepts like ubiquitous programming and micro-programming (cf. my article on ubiquitous programming with pen and paper).

Similar to social aspects, a PL designer can consider didactic aspects. Curl language, for example, was carefully designed to address issues such as a smooth learning curve and avoiding discontinuity spikes. My own efforts address those plus additional didactic concerns: compositional properties as a formal basis for intuition, a smooth progression from UI to PL and back so people gain useful programming intuitions without being aware of them, object capabilities as a means to formally graduate authority of a growing developer, support for revocation and resilience so people can perform exploratory programming and recover from mistakes. (I describe a few of these in a recent LtU comment regarding the motivations for RDP.) Bret Victor recently describes Learnable Programming, which certainly benefits from various PL design elements.

I vaguely remember that I once thought as you seem to - that PL design as something so distinct from various social, didactic, CSCW, and UX aspects. That view is encouraged by many programming languages. But it is not essential. Today, programmers experience a great canyon between PX and UX. But, like most canyons, we can find some places that are easy to bridge, and even find their ends if we walk far enough along their edges. If you seek a smooth live programming experience, I think that leads quite handily to regions of the design space where the distinction between PX and UX is only a fuzzy line on a political map.

I agree that developers will always need to concern themselves with dependencies, relationships, integration. The question is: how much will the PL help? When I consider possibilities such as stone soup programming I think the PLs can help a great deal in addressing various aspects - social and otherwise - of dependency management.

Many thanks!

My science fiction comment was based on your motivations for RDP :)

Is there a need in a large scale system for automatic selection of components? Perhaps. I'm not really convinced of that, but I'm not arguing against it either.

What I am arguing here is that this shouldn't have anything to do with 'import'. Importing symbols should just be about namespace management. e.g. when you write 'import MP3Player', you should just be bringing in symbols defined in that namespace.

Look at your "import by need" idea from your link. It suffers from two additional problems to the ones you have listed:

- It resolves look-up after dev-time via a name. That's bad. Variables should be declared. Name look-up post dev-time is bad.

- You have some ad hoc assertion mechanism that specifies some properties the import is to have and you have to specify them every time you import that function.

Note that even C-style header files didn't have those problems. The main problem we had with C-style header files (ignoring the fact that it was a terrible pre-processor hack) is that they required each declared symbol to have a single global implementation. The solution, IMO, is to instead track such partially specified symbols as parameters and leave imports to namespace management.

Loading symbols or specifications I don't have a problem with, so long as it happens in a structured manner to control entanglement and coupling. My articles modules divided: interface and implement and user-defined syntax each describe structured use of names for loading symbols, specifications, or even full syntax.

I disagree with your "that's bad" assertions. Since you make no attempt to justify those assertions, I am unsure what assumptions you're making... but I doubt I share them. Look-up of behaviors after dev-time by shared symbols is the heart of OOP, row polymorphism, and other disciplines for modularity. More broadly, runtime lookups are quite useful for service brokering, hyperlinking, mime-types and app extensions, mobile code, etc.

That aside, nothing about import-by-need requires it happen "after dev-time". Much can be resolved statically. A language designer can address concerns for enforcing when certain dependencies are resolved, whether at dev-time or at deployment-time or even some later time. So long as we have clear stages, the timing of dependency resolution can be quite orthogonal to the mechanism for dependency resolution.

The problem of specifying on every import can itself be modularized. I did address that in the linked article. I agree that one might more conveniently address it with named interface modules, at a moderate cost to flexibility and uniformity.

Your "modules divided" article seems reasonable and along the lines of what I'm suggesting/doing. You still seem to be coupling import with interface, whereas I think it's simpler to allow people to import whatever they want (interface, implementation, new types, etc.). The important bit is that importing gets you exactly what's there statically at import time -- not also a promise to link to some other implementation. Linking is parameter substitution, and should be an orthogonal mechanism to namespace management.

I don't understand why you say that 'import by need' can be resolved at dev-time. In some cases I'm sure it can, but, in general, implementations need not even be dev-ed yet when you import an interface. What am I missing?

I didn't offer too much detail for why post-dev name resolution is bad other than to compare it to declaring variables. I still won't, because I think the advantages of static binding are well known and the analogy is pretty strong. You say that "shared symbols" are the heart of OOP, etc., but I was careful to refer to "names". The heart of OOP is dispatch on messages, but static binding of names to messages is consistent with OOP, commonly done, and is IMO a good thing. The same applies to row polymorphism or "other disciplines of modularity".

You're assuming we must support a "general case", but you seem to be forgetting that general cases don't happen unless we want them to. A language designer can ensure, for example, that validation tests on import are either pure or constrained in authority according to stage.

Re: You say that "shared symbols" are the heart of OOP, etc., but I was careful to refer to "names".

I noticed your use of names. But your careful use of names ("It resolves look-up after dev-time via a name. Name look-up post dev-time is bad.") was inappropriate when describing the problems with import-by-need, which uses shared symbols but does not use names. So, with that context, I assumed you have not seen or experienced much of a difference between the two (which is not unusual).

it's simpler to allow people to import whatever they want

If by simple you mean uniform, yes. But it's simplistic. It leads to complex entanglement issues. Consider your reaction to the assertion: "it's simpler to allow people to mutate whatever they want".

That said, by 'interface' I'm envisioning something close to an ML module signature, perhaps with some behavioral contracts. Support for types descriptions or a few fully constrained definitions would not be unreasonable.

The important bit is that importing gets you exactly what's there statically at import time -- not also a promise to link to some other implementation. Linking is parameter substitution, and should be an orthogonal mechanism to namespace management.

No, no, no. That is exactly wrong.

The important bit is that importing DOES NOT tightly couple you what is available at import time (whether that time is static or otherwise). This decoupling provides the wiggle room to avoid entanglement, enables orthogonal code upgrades, supports adaptation to different environments or resources, enables users to leverage preferences and policies.

Namespaces are not essential for computation or scalable programming. At best, they're a convenience. Linking of modular components, whether via substitution or unification or some other means, is the essential and important bit.

After reading your other post, I don't think we're in nearly as much disagreement as this new post implies. The most important issue semantically is linking, which I agree can be more than just vanilla parameter substitution (unification or lightweight constraint integration are fine) even though I still think of it as just substitution.

But...

The important bit is that importing gets you exactly what's there statically at import time

No, no, no. That is exactly wrong.

The important bit is that importing DOES NOT tightly couple you what is available at import time (whether that time is static or otherwise). [...]

The mechanism for avoiding tight coupling is linking. You specify that your component has a certain freedom (a generalized "parameter" that could be refined through unification / constraint solving) and then that "parameter" can be substituted/refined at a later link time.

Nothing in the preceding paragraph requires or has anything to do with imports! The only reason 'import' is related to any of this is that library producers will generally deliver an implementation and a specification and library clients will generally import the specification. And the specification should be imported verbatim.

And I see no reason to limit importing any kind of value or type. Any such restriction would just be annoying and serve no clear purpose. It's not like mutability at all in that regard. I guess you could argue that by forcing programmers to only import abstractions you are nudging them in the right direction, but I'd hate such a rule. To me that's just as arbitrary and annoying as a rule that functions can only be 100 lines long. No thanks. Just get rid of the obstacles that would make me reticent to use abstraction wherever appropriate and I won't need any prodding from a nanny language.

As a reason for why I will sometimes want to import an implementation rather than an interface, I believe that you should import enough specification that you can, in principle, prove the correctness of your local component. Finding a useful abstraction that specifies all of the properties a client will need is non-trivial. In cases where you're not going to bother abstracting (no parameterization), it's often better to just depend on the full implementation (at least until you have time to find a good abstraction).

Finally keep in mind the context: I'm responding to your claims that imports are bad. I'm not trying to prove that namespaces or packages must be present in a language. I'm only arguing for the existence of clean approaches to imports (orthogonal to linking) that don't have the entanglement issues you're worried about.

The phrase "correctness of your local component" is a contradiction in terms. Correctness is always contextual, never local. I.e. a function is correct with respect to a type or spec; a type is correct with respect to some larger context. (This is also why correctness is not compositional.) But you can prove partial correctness up to a partial specification and context.

It's not like mutability at all in that regard. [...] won't need any prodding from a nanny language.

Many of your complaints have been uttered before (with similar pathos) by people who don't see issues with pervasive mutability, who believe their personal discipline, experience, and foresight are reliable at scale. They don't like the idea of nanny languages, either.

You say you "see no reason", but I think that's an issue of perspective. Try a different one.

We prevent pervasive mutability for reuse, reasoning, and sanity of downstream developers. Controlling entanglement is similarly in support of downstream developers, who may wish to extricate part of a project or take most of a project code but with a few extensions or tweaks to fit a new context (and hence a slightly different notion of 'correctness'). To that downstream developer, your close-minded efforts to import implementation details is cause for much frustration.

Much of the time, you are a downstream developer. Designs that benefit downstream developers benefit you. Therefore, you have much reason to favor nanny languages that 'nudge' upstream developers, though you may not initially appreciate them when writing your own bits of code.

We have "don't pollute the river" laws because, no matter how much you believe you can depend on your own discipline, you can't depend on the discipline of every industry upstream.

I'm only arguing for the existence of clean approaches to imports (orthogonal to linking) that don't have the entanglement issues you're worried about.

You have yet to present an argument that the importing of implementation details and ad-hoc values and types won't have entanglement issues.

Correctness is always contextual, never local. I.e. a function is correct with respect to a type or spec; a type is correct with respect to some larger context.

So? That you need details of the context was my point. Bind to the details you need. If you don't have an abstraction in mind, bind to the implementation if it's convenient. Yes, having implementation details scattered afar isn't good. So don't do that. IMO the ball is in your court to explain why namespace boundaries are a good place to enforce abstraction boundaries.

I'm going to abandon the discussion of which metaphor is most appropriate. That's getting a little too indirect. Except to say that I intend to prevent river pollution by filtering it out periodically at hydroelectric plants in my language.

So? That you need details of the context was my point. Bind to the details you need.

If you "bind to the details you need" and "you need details of the context", then the result is unreusable, inflexible code - i.e. code that is bound to a specific context, entangled with it.

IMO the ball is in your court to explain why namespace boundaries are a good place to enforce abstraction boundaries.

I did describe how I reached this conclusion in the linked article on modules divided so I did not see any reason to repeat it. I'll repeat the most immediately significant points for the lazy:

• Dependencies between interfaces must be very tightly constrained in their dependencies to control entanglement.
• To support effective reuse of specification code, interfaces may derive from one another and support limited parameterization.

The articles also provide an objective, operating definition of entanglement, in terms of the number of modules that must be copied to use a particular subprogram in a new project (new context).

Interfaces can provide some simple values (primitives, records, etc.), but nowhere near "any kind of value or type" simply because it cannot arbitrarily import other interfaces. But I don't see how implementations can ad-hoc import implementations while still controlling entanglement! I don't consider namespaces a "good place to enforce abstraction boundaries". Rather, I consider them a bad place to bind implementation details or context, for reasons of entanglement.

If you "bind to the details you need" and "you need details of the context", then the result is unreusable, inflexible code - i.e. code that is bound to a specific context, entangled with it.

True, but there's no general way to avoid that. Proving that you're using some value in a way that produces the desired result usually requires additional specification of that value over and above e.g. the HM type. It doesn't need to make the code any more brittle or entangled because the code can be usable even if the proofs are broken. But there definitely seems to be some pressure to postpone proofs to later when things are somewhat stabilized in order to avoid breaking proofs repeatedly as things evolve (similarly, there's pressure to delay optimizations).

Dependencies between interfaces must be very tightly constrained in their dependencies to control entanglement.

To support effective reuse of specification code, interfaces may derive from one another and support limited parameterization.

I don't see anything about namespaces in that list. The reason it's difficult for me to follow the argument from your blog is that your imports do two things, and it's not clear to me which parts of the argument you think would still apply to the factorization I've been discussing.

[Interfaces] cannot arbitrarily import arbitrary other interfaces.

This doesn't make sense in my terminology. Interfaces don't import anything. Importing is namespace management. Do you have in mind that someone could write 'import windows.h' and suddenly everything they write has thousands of dependencies? That's not how my system works, for example.

Rather, I consider them a bad place to bind implementation details or context, for reasons of entanglement.

Namespaces are just for code organization. I should be able to organize my code any way I please. I don't want to be told that I can't split up my code along a boundary because that would violate some arbitrary rule. Organizational units are not abstraction units.

The module system I described leverages constraints to complete the links, I.e. construction by proof. This is a general way to address additional proofs (beyond HM types) without hurting flexibility or adaptability to contexts. (Actually, while enhancing flexibility and adaptability to contexts.)

I described interfaces in the article quite precisely. Why do you substitute your own terminology? Roughly, an interface is a module signature, ML style. It can serve as a namespace, albeit a closed namespace (as opposed to the cross-file C++ namespaces).

Your position that you should be able to organize code any way you please is contradicted if you have a goal to control entanglement. Entanglement is primarily an issue of organization, not abstraction, but I do not find it surprising that organizational constraints impact abstraction (cf. the earlier discussion of services). You've made your position and desires clear, but never justified them as wise.

This conversation is too broken to fix easily. Maybe we can revisit this when I have something that I can share and point to and ask "where's the entanglement?"

Organizational units are not abstraction units.

This seems a rather extraordinary claim to make in passing.

I think if I made a list of organizational units, the vast majority are would directly correspond to abstractions. Consider just a few: directory, text file, paragraph, service, module, DLL, record, function, role, group, account.

Might a more rational position be just the opposite? Organizational units are the heart of many useful abstractions. What is a cloud? an organization of water vapor. What is bicycling? an organization of man, device, and motion. What is a monad? a structure that organizes code to compose in a sequence. Even broad abstractions - space, time, water, grass - get measured, divided, sometimes named into useful structures and organizations.

Entanglement is an organizational issue. I do not find surprising that constraints on organization (e.g. to control entanglement) can impact abstraction. I do find it surprising that you assume organization and abstraction to be separate issues. If you've some great revelation that will make your position as obviously correct as you assume it to be, I'd be happy to hear it.

What is bicycling? an organization of man, device, and motion.

This seems a rather extraordinary claim. Bicycling is a form of exercise and sport! Motion is an abstraction and can't be part of an organization! Women use bicycles, too!

If you've some great revelation that will make your position as obviously correct as you assume it to be, I'd be happy to hear it.

I'm not claiming that I'm going to going to shock and awe you into agreeing with me. I'm just hoping to provide enough context that we can communicate effectively.

I'm just hoping to provide enough context that we can communicate effectively.

Then could you make a habit of justifying more of your assertions? Your arguments tend to consist of assertions I find dubious ("Name look-up post dev-time is bad.", "Importing symbols should just be about namespace management.", "I should be able to organize my code any way I please.", "Organizational units are not abstraction units.") presented without any explanation or justification. I suppose you believe them to be self-evident. When I ask for justification ("I am unsure what assumptions you're making", "Might a more rational position be just the opposite?") I'm simply rebuffed or ignored.

I don't know how you hope to provide any context except through communication. Perhaps you should blog some of your ideas or examples, so that you can link to them for extra context. It won't help for the lazy people who can't be bothered to follow links, of course, but they'd have no right to complain about lacking context.

What's becoming clear to me is that some heavy infrastructure is needed to support all the compile-time, link-time, and runtime functionality you need in order to fully integrate interfaces, contracts, capabilities, and unit tests with modules and imports.

A "types-only interface specification" tells you the HM types of arguments and return values for a set of functions. A "contract" names a set of semantics those functions are supposed to have and a set of capabilities which they are not supposed to exceed. These two together are an "interface" as I'm thinking of it.

The contract is answered by a corresponding assertion in the unit that it conforms to a named contract. Next you have a "unit test" that verifies conformance to the contract for at least a few dozen cases, while profiling the proposed import to see about its performance and to see how much of its code gets exercised by the test cases.

Ideally the unit test reports 100% code coverage (meaning no unrelated functionality is hidden in the unit) and contract conformance. In practice 100% code coverage is hard to achieve when you don't compose the unit test while looking at the proposed implementation, so you will probably configure your system to prefer whichever potential import has the greatest code coverage (and speed) within the unit test, and may extend the unit test by generating "random" cases to test (at the expense of potentially long test runs) to try to get code coverage higher. Alternatively, in full paranoia mode, you may configure your system to "stub out" any code not exercised by the unit test with a jump to an error handler.

The unit tests go with the contract, and may require as much code or more code than the unit being imported! Their only virtue in terms of saving development effort is if the contracts themselves are standardized, and the unit tests may therefore be part of a standard library. In this case an implementation of the functionality could also have been part of the standard library, so, um, why not?? Because maybe the import integrates with something hairy or performs better? So, unit tests may include a known-valid but somehow undesirable implementation of the functionality itself to check the potential import against.

Each potential import may carry with it additional contracts (and unit tests) for subcontractor interfaces/libraries that *it* needs. The capability limitations you specified with the original contract that a unit must fulfil, are transitive and must be applied to these subcontractor units as well. If subcontractor libraries contain code not exercised by your unit tests of the library they're subcontracting for in the case of your program, you may need to auto-stub that code as well. This is because the unit tests for subcontractor units in untrusted code are themselves untrusted; you don't know that they're checking the right thing, so you can't accept their word that something conforms as testimony that the conforming thing is relevant or conformant to your own requirements.

Selecting the units that your build will use means selecting imports such that the transitive closure of all imports starting with those needed by your program is small/manageable/fast/has other desirable characteristics.

Eventually you identify a set of imports that will "complete" your program, and you link them all together. Now you have to compile in the capability management, valid-return assertions, and checks for any type information you couldn't prove statically, that you got from the program and from every subcontracting unit.

If you imported something in full-paranoia mode and inserted jumps to a handler rather than trusting un-exercised code, you have to add the handler, and the handler ought to record the set of arguments that led to testing the unchecked code, extending your unit test. If your unit test included a known-correct but undesirable implementation of the functionality, your handler may be able to jump to that implementation and retry, assuming it doesn't require state information that exists in a form stored by and only available to the imported code.

In all of this, you have to distinguish "trusted" code -- ie, code that someone you have paid money to and have a contract with accepts specific liability for, or locally-developed code that doesn't merit fully paranoid checking because you know you didn't put anything malicious into it, from "found" code -- ie, downloaded-from-somewhere code from someone you don't have any business relationship with, that requires full paranoia even at the expense of runtime checks.

Is this a reasonable description of the infrastructure you need to make this "interchangeable imports via contracts and capabilities" model a reality?

Ray

I've been interested in techniques that might allow developers to express general contracts then use those to automatically generate a mix of tests and local proofs. This allows simple assertions to cover large numbers of tests, while allowing us to move forward with high confidence that might be less than 100%. I've seen systems that are good for proofs, and I've seen systems for automatic generation of tests, but I've not seen a system that provides smooth interpolation between the two.

I think 'auto-stubbing' to remove untested code is an awful idea. Even if code is untested or incorrect with respect to the domain model, it may still be protecting more global properties (type safety, duration coupling, etc.) that are easy to validate or enforce structurally but cannot be generically achieved via stub. Similarly, injecting runtime tests is dubious if the only thing you can do upon failing a test is violate abstractions a bit earlier.

Now you have to compile in the capability management

Why would you need to do this?

Of the capability models I've studied, object capabilities are the only ones worth learning and knowing. Object capabilities describe first-class, tight coupling of designation and authority. Developers must explicitly manage their object capabilities, and explicit management is a significant aspect of their value for secure interaction design - ensuring visibility, awareness, and that the path of least resistance is to grant least authority.

Using ocaps, you don't need to pass tests or type-checks to protect the important abstractions and services. You only need memory-safety, which is easy to ensure generically. Incorrect code is constrained in the damage it can cause by the object capability model. Developers can readily mix trusted and untrusted code.

Even better if your programming language also enforces real-time properties. (Batch processing tasks can always be modeled via incremental state manipulation with real-time increments.) If you have both ocaps and real-time properties, you're additionally protected against denial of service attacks and most timing-based covert channels.

You can have a healthy paranoia without elevating it to insanity.

Is this a reasonable description of the infrastructure you need to make this "interchangeable imports via contracts and capabilities" model a reality?

No. You can do a better job with simpler mechanisms.

Many of my comments that involve 'should' should be read as 'the way I think this should work' rather than as an assertion that I can prove these preferences are optimal. I do, of course, have reasons for my preferences.

Why do I think that unsupervised binding of names is bad? For one, it's a security problem. As I recall, Z-Bo's link has examples of what can go wrong. When I type a name, almost always, I'm attempting to reference a specific entity (even if that entity is a parameter). I want my IDE to make any ambiguity clear, allow me to resolve it, and have it easy for me to inspect what I've referenced. That requires it to happen at dev-time. For the same reasons that it's good for security, it's good for avoiding certain bugs.

Regarding 'organizational boundaries are not abstraction boundaries', I think it's useful to be able to import some but not all of the symbols defined in an abstraction, to rename symbols and give them local aliases, and even to split an abstraction between several namespaces (though that's probably rare). I'm not arguing that this is the one true way (though obviously I prefer it), but I am arguing that I don't see any entanglement issues. You claim you've demonstrated the entanglement issues on your blog, but then you summarize the argument in a way that doesn't mention name importing or namespaces at all.

The entanglement issue that I understand you to be describing has to do with failure to abstract properly, leading to dependencies of abstractions on things that should be implementation details. If names are semantically part of your abstractions then, yes, you can run into entanglement issues from failing to abstract from the names. But since I'm the one arguing that names and organization should not be semantically meaningful, I have a hard time understanding why you think my approach leads to entanglement.

The reason I am tempted to bail on this discussion is that it doesn't seem to be getting anywhere and because I do think that it will be much easier for me to explain my position with a little IDE for my language and some documentation rather than trying to figure out what set of concepts we're not using the same words for.

There are insecure approaches to binding of names. There are also secure approaches. It isn't clear to me that `unsupervised` is the relevant distinction. (If you depend heavily on supervision for computer security, you're doing it wrong.)

Use of names for 'specific entities' seems either rarer or less specific than you've been implying. For example, it is unlikely that you're using version numbers or hashes in your names. But I agree with a goal to understand and control ambiguity and resolution, both at dev-time and run-time.

Where you speak of current practice ("When I type a name, almost always [...]"), I note that you're constrained by current languages. It's difficult to appreciate other possibilities without experiencing them. I expect your habits would change if you had more options readily available.

The entanglement issue that I understand you to be describing has to do with failure to abstract properly

Entanglement is strictly an organizational issue. Entanglement may be objectively measured by the following question: "I want to take this syntactically bounded subprogram from project A and install it in project B. How much code do I need to haul to project B for this subprogram to be complete in its meaning?" Entanglement nearly always exists. The issue is controlling entanglement (e.g. with a big-O constraint or an asymptotic bound), not preventing it.

Entanglement isn't about abstraction. However, organization has a significant impact on expression of abstraction. It is necessary to constrain organization to control entanglement. Those constraints WILL impact abstractions.

Your "importing symbols from namespaces" does not control entanglement in any way. At the very least, you'll end up entangling the code that lists symbols from each namespace.

It isn't clear to me that `unsupervised` is the relevant distinction. (If you depend heavily on supervision for computer security, you're doing it wrong.)

Well, it seems we agree on the problems and some of the goals. This subthread started with my criticism of your proposed technique of import by need. That criticism was based on a particular understanding of what you had in mind, that perhaps was not correct. If that's the case, then I don't mind retracting my criticism, but I still haven't understood the details that would lead me to believe that's the case.

It's difficult to appreciate other possibilities without experiencing them.

It depends, but it's not that difficult in my experience. It's difficult to appreciate them without understanding them, though.

It's difficult to appreciate other possibilities without experiencing them.

I don't have trouble understanding this. I understand that I'm speaking to my personal viewpoints, even if it's not always clear, for expedience, in every sentence. You're the one that blogs about the non-existence of whole classes of solutions. When I speak up that I think you've missed a possibility in the design space, you insist that I consult your proof to the contrary.

For example, it is unlikely that you're using version numbers or hashes in your names.

Usually when I'm developing a module, the specific intention I have in mind with a symbol is "the most current version of this construct". Occasionally, I do intend "thisspecific version of this construct", and support for that is in the works.

Entanglement may be objectively measured by the following question: "I want to take this syntactically bounded subprogram from project A and install it in project B. How much code do I need to haul to project B for this subprogram to be complete in its meaning?"

In my system you will be able to export precisely what you need, regardless of how things are organized. And what you need is strictly about how well you've abstracted the dependencies. I have never claimed importing symbols from namespaces addressed entanglement - merely that it doesn't cause it.

Precise control of export is valuable. It can prevent accidental bindings. But I've yet to encounter or design a PL or architecture where it appreciably controls entanglement. Relevantly, programs become entangled due to lots of fine-grained, short sighted, intentional bindings.

"the most current version of this construct".

Oh? And do you mean that globally? (Is there even a total ordering on versions?) Or is there also some implicit constraint on where you look for this version? And where do half-completed versions fit?

My intended approach is actually more precise and formal about the sort of linker magic traditional languages leave to ad-hoc tooling. My motivation had a lot to do with understanding and modeling names (esp. Nominative types) in open distributed live programs where multiple versions must coexist.

Oh? And do you mean that globally? (Is there even a total ordering on versions?) Or is there also some implicit constraint on where you look for this version? And where do half-completed versions fit?

-- Version 1 source:
x = 1 
y = x 

-- Version 2 source:
x = 2
y = x 

a = y -- a = 2
b = y@ver1 -- b = 1

Syntax is made up. The point is that no annotation means "the version that exists in the current version", but there is some way of bringing in older versions from source control.

I would be much happier if you were to take no annotation to mean 'automatically annotate as requiring the version tested against'. 'Current' or 'latest' version is far too ambiguous, far too dangerous.

Any open-ended handling generally suggests ambient authority, which will either be abused, misused, or hard for programmers to understand the effects thereof. See a related point I made a long time ago: A little harder to get right than you might think.

But imports are fine just so long as you remember they are only for interaction with people (to aid with binding resolution).

A crucial innovation of bondi and its underlying Pattern Calculus is that the pattern variables used for reduction are separate from the binding variables used to control scope. In terms of usability, how would you say the various techniques in Haskell grade out for "interaction with people"?

Imports are fuzzy. Imports should be dev time.

Can you give me a practical example of what you mean? Does it mean the system allows a request whenever a subject has some capability that would satisfy it? Or does it mean the subject is required to explicitly present the capabilities that it wants to use with each request?

Consider my question in light of how you might implement configuring a system's run-time implementation.

I found your clarifying follow-up useful, Z-Bo.

Can you give me a practical example of what you mean? Does it mean the system allows a request whenever a subject has some capability that would satisfy it? Or does it mean the subject is required to explicitly present the capabilities that it wants to use with each request?

What if the answer was the subject is required to explicitly present the capabilities that it wants to use with each request?

Build the system configuration out of rights amplification primitives, such as sealer/unsealer pairs and sibling communication. Alternatively, you could use an EQ operator, but using EQ is antithetical to this whole subthread since it ruins developing a good equational theory for a language.

Also, encourage developers to learn sound patterns for secure configuration, such as Grant Matcher.

Another big implication is that anything globally accessible must be transitively immutable.

anything globally accessible must be transitively immutable

In live programming, we must acknowledge that the entire program is mutable. Every function, constant, module, etc. is mutable, and we do access those. But authority to cause this mutation should be pretty well controlled - e.g. in many cases, it is not and should not be an authority held by the program itself. So I'd tend to weaken that constraint a bit. Try, instead: "authority for mutation must not be globally accessible".

A capability-secure program can be deeply mutable. But there must not be any hidden channels for communication between subprograms.

if there's any chance that a service will be chosen without human input -- then you're going to get malicious services

To be fair, human input doesn't really make a difference. Humans lack any real ability to inspect any code for obfuscated malicious intent. It's hard enough to spot accidental malign bugs and security holes that nobody is trying to hide. A human will do what humans always do: favor some sources over others based on who they decided to trust a long time ago (which updates slower than it should due to decision fatigue). In terms of code, that might be based on signatures or shared private registries. If we want to express anything stronger than a weak or moderate trust preference, the onus is ultimately on the machine.

Eliminating ambient authority is certainly useful for controlling and reasoning about what an import can do. I depend heavily on object capability model and a few simple patterns (involving fine-grained registries) to control what an import can do.

Proving any theorem about code will tend to eliminate both malicious and buggy code. As Benjamin Pierce notes in Types Considered Harmful, it doesn't much matter which theorems you attempt to prove. So proof-carrying code is also a great way to ensure safe and secure code. Type systems, linear types, etc. are ways of providing lots of small theorems to prove.