Lambda the Ultimate

I'm trying to make a research compiler which is supposed to run as a service, and so I'd like to use some language with cooperative, userspace threads, as (Go or) Erlang. The problem is that some user requests may consume too much memory, or possibly never halt. So I'd like to impose some restrictions, e.g., "requests from IP a.b.c.d may only use up to 80mb of heap memory and run for 5 minutes top".

Haskell has setAllocationCounter (also forkIO and STM), which would help, but that doesn't seem to take the GC into account. Erlang seems to let me limit the heap for each process, but many tasks will be computationally intensive, and Erlang's probably not a good pick.

Would anyone have some suggestion? I'd rather not have to create a new language, but it's a possibility; is there any research on such languages that could help?