Home » forums » LtU Forum

The Memory Pool System: Thirty person-years of memory management development goes Open Source

Not strictly language related, more language implementation related, but worth bringing to the attention of LtU. Quoting from the abstract to this paper:
The Memory Pool System (MPS) is a very general, adaptable, flexible, reliable, and efficient memory management system. It permits the flexible combination of memory management techniques, supporting manual and automatic memory management, in-line allocation, finalization, weakness, and multiple simultaneous co-operating incremental generational garbage collections. It also includes a library of memory pool classes implementing specialized memory management policies.

Between 1994 and 2001, Harlequin (now part of Global Graphics) invested about thirty person-years of effort developing the MPS. The system contained many innovative techniques and abstractions which were kept secret. In 1997 Richard Brooksby, the manager and chief architect of the project, and Nicholas Barnes, a senior developer, left Harlequin to form their own consultancy company, Ravenbrook, and in 2001, Ravenbrook acquired the MPS technology from Global Graphics. We are happy to announce that we are publishing the source code and documentation under an open source licence. This paper gives an overview of the system.

(emphases mine) Good news for language implementors! Richard Brooksby gave a talk on the system at last night's SchemeUK meeting, and it sounds like it has a few killer features that would make it worth considering over a roll-your-own solution or an embedding of, say, the Boehm-Demers-Weiser collector.

(One interesting nugget of information he mentioned was that during development of the MLWorks compiler and system, before they started development of the MPS, they measured the cost of object allocation at two-and-a-half instructions per allocation, on average - two instructions for the allocation, and half an instruction for the garbage collection overhead! I don't think MPS is quite that efficient, but it certainly has a lot of potential.)

By tonyg at 2004-12-16 17:04 | LtU Forum | previous forum topic | next forum topic | other blogs | 13715 reads

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Previous mention

Memory Pool System back from the dead

By Anton van Straaten at Thu, 2004-12-16 19:10 | login or register to post comments

Is there anything wrong with the GC package?

Hmm, I sometimes program around a bit on my own PL (after a year i am now going for typechecking; too many other hobbies), and uhm, just chose C++ with the Boehm et al conservative collector as my target language.

Is that a bad choice?

By marco at Sat, 2004-12-18 13:15 | login or register to post comments

Runtime Target Choice

Marco: Hmm, I sometimes program around a bit on my own PL (after a year i am now going for typechecking; too many other hobbies), and uhm, just chose C++ with the Boehm et al conservative collector as my target language.

Is that a bad choice?

Not necessarily, but probably. I would strongly recommend you consider targeting either C-- or LLVM instead, as both are considerably more flexible in their support of things like GC, exceptions, threads, tail-call optimization, etc. than C or C++.

By Paul Snively at Sun, 2004-12-19 20:02 | login or register to post comments

target languages

Well, IMHO targeting C (or C++) is really the most practical alternative. C-- doesn't appear ready for real use - it's awkward to build from sources, since it has lots of external dependencies (mk, noweb, ocaml, ...), it is still in a prototype phase, and in several ways is much less flexible and target specific than (say) C. Tail calls are supported, but what C-- calls continuations are just non-local exits.

LLVM isn't that bad, but it doesn't offer tail-calls and/or continuations (yet).

So I would recommend C: it is rather liberal in what it accepts (especially gcc), it is *much* more portable than any existing alternatives and with a bit of hacking it is possible to shoehorn quite of lot of things into it (again gcc comes to the rescue with stuff like computed goto's).

Besides: I doubt the C-- or LLVM maintainers can keep up with the amount of tuning and porting that is put into gcc.

(IIRC gcc 4.0 already handles indirect tail calls)

By Bunny_351 at Mon, 2004-12-20 12:32 | login or register to post comments

I agree

Those were my arguments too. As far as I read C-- doesn't produce very fast code for the moment, and LLVM doesn't seem mature (I might be wrong there).

Also, I am kind of hoping that I can stay away from generating assembly for some target machine and the latest gcc hacks seem to make C and C++ a great target language (optimizations on one time write variables, register allocation, tail recursion, etc.)

But then, I am at the start of semantical analysis, so...

By marco at Mon, 2004-12-20 18:24 | login or register to post comments

Fine for messing around

If you are just messing around then anything that gets the job done is fine. If you are creating a language implementation that is expected to run, e.g. business software, where the cost of failure is high, the memory leaks that conservative GC may bring about cannot be tolerated.

For example, PLT Scheme by default uses Boehm and suffers from memory leaks as a result. This normally isn't a problem, but if you attempt to use the PLT web-server on a long running or high load site you will run out of memory. There is a precise GC in development and I have mentioned the MPS to them after I attended a talk about it at Scheme UK.

By Noel at Mon, 2004-12-20 13:40 | login or register to post comments

True

I am just messing around, that's for sure. Still, even when messing around I like to make the right choice.

From what I know the Boehm GC works fine until a 100Mb of data (I think I got this from sheets). The PLT scheme GC problems I hadn't heard of and generated a lot of hits on Google. Thanks for the hint.

By marco at Mon, 2004-12-20 18:47 | login or register to post comments

Definitely front page material

I had just thought to post the link to the front page, when LtU search turned up this forum topic. It should be there, the sooner the better!

By Charles Stewart at Tue, 2009-11-17 10:30 | login or register to post comments

re: front page

Given the age of the article, we would definitely be better off posting a new story about MPS instead of promoting this to the front page directly. Last time we promoted a story it appeared on the front page but below newer articles, making it easy to miss, but this would get totally buried.

By Leon P Smith at Tue, 2009-11-17 11:18 | login or register to post comments

Anyone using this?

Thanks for mentioning Memory Pool. It looks interesting, but I'm wondering why it seems close to abandon ware. For instance, the latest release is a few years old and has no built in support for any x86-64 even though there is older support for 64 bit platforms like the Alpha, suggesting nobody cared enough to update the distribution. The older LtU discussion suggests that Memory Pool would be a theoretically better choice than Boehm due to fewer memory leaks, but is there another side to the story?

By Josh Stern at Wed, 2009-11-18 18:09 | login or register to post comments

Dylan is using it

Apart from Global Graphics, which as far as I know still use it, the Open Dylan compiler (former Functional Objects, DylanWorks) still uses it. The MPS group and the DylanWorks group have worked together at Harlequin to develop a modular, pluggable garbage collector for DylanWorks.

By hannes at Wed, 2009-11-18 22:12 | login or register to post comments