Lambda the Ultimate

Nice collection of links, and well worth looking at, but there is support for dynamic linking in dynamic languages ... maybe not as widely used as it should be.

Ruby has long had support for dynamic loading of C (and other languages) in scripts. See for example thew articles:

• Ruby Inline introduction
• more on Ruby Inline

Perl also has inlining and dynamic linking (and has for a long time). I'd guess that other languages do to, but I'm not sure which ones or how they support it.

One pragmatic concern is that dynamic linkers are notoriously platform-specific. The three major desktop platforms all use different binary formats (PE, ELF, Mach-O), and differ significantly enough in implementations that it's difficult to write an abstraction layer. Even amongst platforms that share the same executable format, the linkers for each OS are very different.

So, unless you're happy to constrain your language to just a single platform, it'd be pretty unwise to build on top of your OS's linker for all your dynamic needs.

Even though I love to bash Microsoft on a regular basis--an activity which seems to have become passe of late--the .Net Common Language Infrastructure, which does much of its work at application deployment/start time, seems to have been utterly ignored by this post.

The Unix linkers (and Windows, of course, has binary linkers similar to those found on Unix systems), are still primarly meant to piece together binaries, and do that job well. However, ld.so and friends suffer from numerous issues which limits their applicability to higher-level languages, especially when dealing with things that are not object files.

Low-level linkers are generally type-unaware, other than trivial type distinctions like "code" and "data"--type information can be encoded in a way the linker understands via name mangling, but that's best understood as a magnificent kludge. :) And it took years of complaints from C++ users, who need things like weak symbols and such to reasonably support features like templates, to even get the Unix linker to where it was--for much of the 1990s, it seems, certain Unix system providers dismissed the concerns of C++ users as irrelevant.

In short--there's a reason many high-level languages (or language runtimes/environments) don't use the system loader in the manner you suggest.

Maybe I do not understand the claim correctly, but there is a lot of dynamic linking and loading going on in modern programming languages. In addition to the inline options mentioned by pate, here is what I found at a casual glance:

• Python ctypes foreign function interface, part of Python standard library
• Ruby FFI library
• Perl DynaLoader, part of core Perl

... and that's just scratching the surface. So again - maybe I'm unclear on what's being said here. I'll have to reread the links given.

From its web page, Alien is a Foreign Function Interface (FFI) for Lua. An FFI lets you call functions in dynamic libraries (.so, .dylib, .dll, etc.) from Lua code without having to write, compile and link a C binding from the library to Lua. In other words, it lets you write extensions that call out to native code using just Lua.

It uses the great libffi library by Anthony Green (and others) to do the heavy lifting of calling to and from C. The name is stolen from Common Lisp FFIs.

Languages could easily take far more advantage of dynamic linking. But focus on ELF binaries and dynamic-loading down at the low-level code is, I believe, a harmful distraction.

More interesting is the issue of how to integrate dynamic linking with languages in an accessible, portable, flexible, and implementable manner. This requires that the language be developed with some standard abstractions that can leverage dynamically linked services, and that the dynamically linked services and virtual-machine be developed in combination to provide these abstractions. The simple 'foreign function interface' concept really isn't enough.

I have my own ideas on that subject, but they do heavily involve dynamic linking infrastructure available to the VM.

I believe that having a "live services" layer - that is, services properly abstracted as either preceding the application going live even without taking any action, as well as persisting across application upgrades - is important and useful for languages (be they static or dynamic), and that some of the better ways to achieve this are through 'service' registries that allow one to perform service matchmaking (i.e. "I require http access" matches "I provide http access").

Services can be provided from many sources - including plugins (e.g. ELF binaries), but also including CPAN databases and such. These services can be heuristically matched and filtered based on policies and requirements, and may dynamically support automatic "fallback" should a service become unavailable (i.e. because its dependencies fail). Assertions and unit-tests can be part of a service to ensure it seems to be live and sane... those could be tested at startup and periodically, especially if their side-effects can be confined.

Live services can also extend to helping hook into live dataflows and domain-addressed multicast networks, as source or sink. Further, one is may also be given capability to automatically publish one's own service for further matchmaking.

I posit that a carefully developed and effectively integrated 'live service layer', implemented with support from dynamic linking infrastructure, could eliminate much boilerplate code, eliminate most need for singletons and abstract constructor or dependency-injection frameworks (a singleton becomes an 'external service', and external dependencies are managed globally), and provide greater robustness in the face of dynamic upgrades of the environment.

Common support for persistence, such that "connections" and objects developed over the life of an application can also be resurrected (unless the developer volunteers for volatility), would probably cover even more boiler-plate and robustness. With care, it would allow switching dynamically to a new plugin without resetting the whole application. This would need to integrate with plugin-provided services and other 'live-services' in a consistent manner, such that the language knows how to resurrect services without duplicating them by accident.

My own approach has been providing these live-services in three layers recognized by the runtime, capturing different persistence, security, and distribution concerns:

• application layer provides 'volatile' services. Volatility is contagious: if A is volatile, and the demand for service B depends on A's survival, then B is volatile. For robust persistence, higher layers can only be volatile via contagion.
• local layer provides 'persistent' services common to the machine or local-area network. These will recover with their previous state. Applications are also free to provide local services (i.e. to automatically re-establish connections or display). Example services include local filesystem, joystick access, local print service
• global layer provides persistent services that are pervasive among machines and have common semantics. This is a security layer for distributed or migratory agents. Services in this layer are available to remote code. Examples would include clock, timers (e.g. once-per-second event), random-number generators, an RSS feed.

Calls from higher to lower layers are possible, but would be capability based: the communication from high to low layer occurs over an opaque, unforgeable connection or name, and so must be part of the state or constructor for the high-layer service.

Services such as scripting engines might exist in more than one layer. For example, javascript could be provided at both local and global layers, providing access to more application services in the lower layer.

@msimoni

It is true that the *nix based systems support linking, but "well" is subjective. Linux is optimized for static linking, and any run-time linking costs are immense.

Awhile back, compiler writer extraordinaire Walter Bright effectively challenged a kernel developer (Andreas Erickson) to fix this problem and I don't believe anything came of it. This challenge was right around the time Walter announced Digital Mars D v1.0.

(Metaprogramming at runtime is perilous, as it is easy to mix up phase distinctions, something we can expect newer dynamic programming languages to discover in a decade (of course we don't know which decade.))

If I had a first law of metaprogramming, it would be that every decade somebody re-invents Stephen Mellor's Executable UML work. In particular, proper object-oriented analysis and development of Mellor's Specification Object pattern seem applicable here.

In my humble experience, metaprogramming is perilous only when the meta requirements change. For example, if you suddenly need a metalogical system in your query language, then MetaLINQ (and therefore LINQ) is fundamentally the wrong starting point. For an unbeatable discussion of this, read Manuel Clavel's Reflection in Rewriting Logic: Metalogical Foundations and Metaprogramming Applications book.

Chicken Scheme makes quite heavy use of dynamic linking by default.

Chicken 4 (now with hygienic macros in the core!) compiles units to FOO.import.scm (containing macro stuffs required at compile time to use this module) and FOO.so (or .dll or .dynlib or whatever) with the runtime code in.

Static linking is possible as an option, but the default is dynamic.

If you (load "foo") and there's a foo.so available, that's dynaloaded and its entry point 'run' to execute the load-time semantics of foo; if there's a foo.scm instead, then it's evaluated, with practically identical consequences (but less CPU usage if there's the .so :-)

Unfortunately, UNIX hasn't won everywhere. The D programming language, being natively compiled, needed some way to load dynamic libs on Windows in a way that would not suck. Because static RTTI structures govern e.g. how dynamic casting and exception handling work, there was a problem of them being duplicated across the host program and DLLs. The issue with the latter is that they may contain no weak/unresolved symbols that are then pulled from the host app at runtime. As a result, using DLLs with D is a massive fail. SO is fine, but could get better still.

The DDL project took on a drastic route - implementing a complete dynamic linker that can load object files and static libs. The bad news is that the main branch is abandonware and only supports one object format (a DigitalMars flavor of OMF). The good news is that the concept actually works pretty well. I've employed it along with a custom modification of the linker for a system that can dynamically compile and link a D module. It's also easy to check for any changes in the source file and dynamically reload the module. The GC will handle the actual unloading so there are no issues of crashes due to pointers dangling into the obsoleted modules. I can basically say: 'linker.load("plugin.d")' and then e.g. iterate through all classes inside the plugin that implement a specific class or interface, then instantiate some. AFAICS, you don't get to enjoy this sort of introspection when using libdl.

Something like DDL could probably be done for other languages and systems, but having a GC is a major plus. On *NIX, libBFD could probably be used to extend the support for other object file formats, if you're fine with viral licensing.

How to Write Shared Libraries by Ulrich Drepper

Modern dynamic linking infrastructure [...]

I think it's interesting that dynamic languages make very little use of the dynamic linking and loading infrastructure provided by modern free Unixes such as Linux and the BSDs.

Linux switched to ELF almost 15 years ago. Much of a human generation. So ELF itself doesn't seem very "modern". Except compared to the PDP-11 a.out it replaced. Though if programming language evolution has been a glacial "looking forward to eventually having all good features from the 1980's", linker evolution has been even more crippled.

But playing linker games, and altering shared objects' global offset tables, in the implementation of dynamic languages, does seem to have not caught on. So now if it does, I guess that use will then be "modern".

Lots of languages use libdl. Goo (dead), for example, was built on it (to eval, it emits C, then compiles and dl links it). And Bellard's libtcc (now again undead:-) lets you compile and link C without touching disk.

But why the long delay in playing more interesting games with dynamic linkage?

One factor has been the technical and social characteristics of the gnu compiler chain. Perhaps even a primary cause. At least in the late '90's, my recollection is the roadblock was a triple of () no api for GOT writes, () gcc folks' "we can't imagine why anyone would want to do that", and () gcc/ld "ball of mud build system" entanglement defeating multiple efforts to create an independent gcc-compatible linker (to improve gcc linker tech, which even 15 years ago, was far from cutting edge). So you couldn't do it in gcc/ld, and you couldn't escape them. Several folks tried to break through, but it never worked out. I've no idea what's been happening this century. Focus has shifted to vm's.

Just so you fully appreciate the self-inflicted wound-ness of this all, here you have a little hash table from strings to function pointers, pointers you are jumping through anyway for any function not statically linked, in writable memory, and you couldn't easily change the d*mn pointers. After all, why would anyone want to write to a hash table, or wish to redefine a function?

Maybe things have now improved?

Thanks for the many insightful comments! You've convinced me that this is the way to go :)