How to load/interpret STATIC libraries at runtime.

I have always been wondering how a language interpreter would run code that requires that a static library be linked. There must be a way of doing it - all the neccessary information is in the library. Thus there must be a way of doing this in pure code.

Can anyone help on this issue? I have searched the internet many times over the course of a few years and have found nothing.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

This can be done...

... but there are a few caveats. For example, linking with code that uses a complex name-mangling scheme isn't fun, nor is tracking down all the dependencies, and maintenance efforts to keep your loader up-to-date with various generations of object file formats will be a pain.

Anyhow, what you want to look for are: object file formats (Wikipedia: Object File, Wikipedia: Object Code), especially ELF (Linux) and Portable Executable (which is used for Windows DLLs and EXEs). Portable Executable seems to derive of the earlier COFF.

[Added:] While you might not think it, many 'static library' and object files essentially use the same file format as the dynamic libraries. A 'mylib.a' static library file in Linux is often an archive of ELF-formatted '.o' files, with ELF being the same format used for shared object '.so' files. Linux offers tools such as 'ar' and 'objdump' to peek inside these files, which might make that platform more suitable than others for exploratory work on a static library loader.

To get native code execution speed, your implementation language will need to allow you to set your execution pointer into a user-provided array or memory mapped file structure. You may run into trouble if your implementation language doesn't support the same execution model as the library code - e.g. stack model vs. continuation passing style, support for exceptions, the direction the stack grows, etc. That is, if you're crossing language boundaries, you'll run into every problem anyone implementing a Foreign Function Interface will encounter.

Also, I don't believe this really qualifies as an LtU topic...

Also, I don't believe this

Also, I don't believe this really qualifies as an LtU topic...

Seconded. (Unless I am missing something)

Oh really?

I didn't realize there was qualification criteria. My fault. I thought this site was for general stuff related to programming langauges. Is it more specific than that?

Yes, really.

Please review the FAQ and policies document, especially the sections on purpose and policy enforcement.

If you think your question is relevant for LtU, it would be a good idea to attempt to explain why.

Ok

Well I read over the policies and while I realize my question isn't about programming language theory, I think it still has relevance because writing an interpreter for any programming language that can use object files requires some way of executing object files at runtime.

Beyond that, this is a place of learning about programming-language-related topics in general. As the FAQ says "we allow ourselves moderate forays to bordering issues like programmability and language in general."

Is there a place on the internet with more intelligent and knowlegable people who know about programming than Lambda the Ultimate? If so let me know and I'll go ask this question there instead.

It is a programming issue

Well I read over the policies and while I realize my question isn't about programming language theory,...

LtU is not exclusively dedicated to PLT - although it is a fascinating subject and occupies a good amount of our time.

I think it still has relevance because writing an interpreter for any programming language that can use object files requires some way of executing object files at runtime.

People who write interpreters also generally have to know how to use text editors and operating systems, but questions about emacs and Linux/Windows would be out of place. Although linking is involved in most compilers, that magic usually occurs outside of the context of the Language. For example, one can make external libraries accessible through such things as Corba, SOAP, etc... But none of these external protocols would be considered directly on-topic, unless they have properties that directly effect the syntax or semantics of a programming language. There is a certain amount of latitude we allow ourselves, but the general rule is that this is directly proportional to the amount of contribution (and goodwill) built up by the person to the community. A wide latitude is generally not afforded to first time posters.

Is there a place on the internet with more intelligent and knowlegable people who know about programming than Lambda the Ultimate? If so let me know and I'll go ask this question there instead.

LtU is not The Programming Weblog or The Language Programming Weblog - programming is but an adjective to the concept of Language. There are a plethora of websites dedicated to programming, but there is no implied fiduciary responsibility of LtU members to provide you with non-PL related pointers. For that matter, any PL help that one receives here is based exclusively on the discretion of the community.

Thanks Chris, for your

Thanks Chris, for your detailed response. I am not a first time poster, but I am new. In general, I find the backlash against "off-topic" posts on these types of sites to be ridiculous and childish. If I posted a topic about pokemon, feel free to slap me. But if I posed an at least semi-related, good faith topic on a rarely (if ever) discussed issue - I do not expect or appreciate the backlash. Not to say you are lashing out chris, your post was very helpful. But a community doesn't grow by allocating a significant amount of its time biting newbies.

I hope my first statement

I hope my first statement didn't come off as 'biting the newbie', but I must say I'm tempted to gnash and gnaw a little now, after being indirectly called 'ridiculous and childish'.

Static libraries and the explicit resolution thereof (as opposed to implicit dependence on a runtime-provided linker) is a topic of potential interest to many LtU readers. That isn't the problem.

The problem is in the first two words: "How to". LtU isn't a How To site. 'How do I load/interpret bitmap file headers?' or 'Where do I look in an MP3 file for the metadata?' and even 'How do I load/interpret Lisp files at runtime?' are only tangential to concerns of language and language theory.

[Admin]

frencheneesz, this is enough. Please review the site policies which served a very large community for quite a few years, and which were thought over carefully. Your posts will be moderated until further notice.

I direct your attention to the phrases beginning "it is firmly suggested..." in the FAQ. Please also change your account information per the LtU policies regarding personal names or identifying information. Your attempt at a sincere question (as you did in your next post) also raises a feeling of disingenuousness given your tone in the current thread.

P.S

David, your original reply was a helpful answer in my opinion, both substantively and as regard what's appropriate for LtU. Thanks.

Programs all the way down

The dirty secret of Programming Language implementations is that they are just programs - PL programmers make programs that other programmers use. A PL implementation is a program that takes input, applies some function to it that produces some output. This process of I/O translation can be formally described Operationally, Axiomatically or Denotationally.

The other dirty secret is that no matter how high level a PL strives to be, buried in the implementation is probably some C code (or worse Assembly). Even those compilers that are self compiling (meta-circular) have to start with a least a minimal amount of hardware addressing, though some later manage to kick the ladder away.

From a PL standpoint, the act of digesting something like text and outputting compiled code requires that the PL (or a foreign interface to another lower level PL like C) be able to interface to the hardware - usually via memory mapped I/O. Once you have access to hardware resources (memory, files, etc...), then you are free to read any programs, lex, parse, evaluate, compile, and then output the results. There are a plethora of algorithms and tools which are available to perform such mundane chores. There are also a lot of official as well as arbitrary standards about linking files together to create executables.

But by using the terminology of linking and static in the question is a dead giveaway that you are interested in a particular type of implementation. For example, Java compiled code does not statically link - it all compiles to an encoded class file (my Java Class File Disassembler comes to mind. Other compilers do systematic global optimization, so they just lump it all together.

The bottom line for your question is that you could look at a multitude of PLs to see how they were implemented. But LtU would probably only be interested in the question of implementation of a particular PL if there were some reason that the implementation strategy had some sort of impact on the resulting language.

Programming Languages all the way down

Keep in mind that there is a corollary to your statement. Everything from Haskell to C to Assembly to Machine Code is a program written in a programming language, though not always one we'd favor writing code in directly.

I wouldn't mind looking at static library formats from a 'file formats are languages too' perspective. The fact that these languages are often used as the output when implementing other languages is somewhat moot. C has properties all on its own whether or not one is compiling Haskell to C. Object file formats have properties worth examining whether or not one is compiling C to object files.

Further, 'linking and static' are issues even in 'live' programming environments and language design (as you might see in Smalltalk or Mozart) because one is often dealing with 'immutable' text-as-code and other values. My perspective and understanding of 'language' tends to encompass the entire Operating System. Runtime executable formats with relocatable code sections are languages that aim to achieve useful properties for reading by the CPU and writing by the dynamic linker and manipulation by the Garbage Collector.

So I don't have any objection to the subject of static libraries.

The only objection I have is to the 'how to'. The task of implementation is outside the scope of LtU.

Data Structures on the way back up

The OP is really asking what data structures are used in the processing (algorithms) and output (file formats) used by PL implementations. My limited understanding of PL implementations leaves me with the feeling that they are faced with the same sort of software engineering crisis that the rest of the programmer community faces. Which probably means that the data structures are antiquated with much rot and not ideally suited for the task at hand. But like any software effort they have to get the job done under less than ideal circumstances.

I would also note that what is considered to be a relevant to the discussion of PLs can be somewhat time sensitive. Back when PLs were first being developed, it was hard to separate the discussion of the environment used by the PL with the PL itself - and to some extent that still holds - witness the conflation of Java the Language with Java the API with Java the Virtual Machine.

Anyhow, I don't disagree with your assessment about why the original question might be off-topic. But I figured actually discussing the relationship (or lack thereof) between PLs and their implementations beat simply engaging in a yes-no contradiction sort of argument. My problem with the original question is that it is expressed in a form that can not be objectively answered. Every PL implementation uses different techniques to solve the problem. The better answer to it would be to say that the OP should pick some languages they are familiar or interested in, and see if they are documented in those projects.

Links please?

Chris: Neither of the weblogs you mention shows up in Google, so if these are references to real, existing things, could you provide links?

I did have the thought...

...to direct all such queries to the BitC mailing list.

[And since this thread is the sole google entry for "The Language Programming Weblog", you might expect a fair amount of traffic]. :-)

It would be welcome there...

BitC certainly needs to solve the underlying problem, and we do use our mailing list for design discussions, so the question would be perfectly acceptable on the BitC list. The only caveat is that we would tackle it as a "how to do this in BitC" discussion, and leave it to frencheneesz to adapt/apply it to his work. If frencheneesz wishes to take it up there, he is certainly welcome to subscribe and ask.

Ehud's comments and the policy document actually led me to some concern about my own postings, since some of them have leaned heavily in the pragmatics direction. I hope that they have been justified on grounds of novelty. I have tried to comply with what I took to be the spirit of the policy, but I am not convinced that I have always succeeded.

That said, there is a real need for a Wiki somewhere that collects implementation wisdom. Too much of that knowledge is not documented outside of its implementing code, with the consequence that language experimenters spend far too much time re-inventing it. a "PL Runtime HowTo" wiki seems like a potentially useful thing. Does one already exist?

As most regular members know

As most regular members know we are not too strict in applying the policy. If something raises concerns someone is bound to mention it, and if the poster is able to explain why the topic is of interest, the discussion usually goes on. If no one complained, it means that the posts were deemed ok by the community.

No David Barbour, I was not

No David Barbour, I was not refering to your first statement as "biting the newbie". Your comment was very helpful. Thanks. [Rest of message deleted by moderator.]