Lambda the Ultimate

Analyzing speed and cause of time loss tends to be deeply complex, and heavily burdened by lots of context (of which the most obnoxious, in typical application scenarios, is often i/o and convoluted dependencies on i/o -- reasoning about server network effects can be really baroque).

Just trying to walk through all the potential provisos first can put an audience to sleep. Here's an entertaining line of questioning you can throw at some server wonks (I'm in the mood to go on a wonk binge since Box likes the word): ask them if they can guarantee (or guess whether) a server will slow down the acceptance of new requests when the backend gets bogged down on something that eats up time and resources. That leads to entertaining conversations, if they care about the answer.

Anyway, I'd hate to attempt comments on why a server in a given language might be slow. I expect servers to generally have lot of trouble keeping the high cost of locks under multithreading in line, while balancing all the other complex demands. Python would need some pretty crisp scheme for avoiding lock traffic jams, and it might not have as much to do with the language per se, so much as a particular implementation. Don't hold me to this paragraph; I want pretend I completely resisted the temptation to kibitz on performance.

More fun is pretending to read Don Box's mind, so let's do that. :-) I'll put on my Johnny Carson mind reader hat and concentrate. Yes, the clouds are lifting. This is what Don Box was thinking when he said he could guess, but would rather be told:

"Yield probably maps directly to a primitive CPS mechanism at the most basic level of the Ruby machine, so yielding might be as close to jumping to code without extra overhead that you can get. It might be like 'return' in C in terms of lack of fuss."

But if I were a Ruby expert, I'd worry that full disclosure might be preliminary to an open ended dissection with lots of opportunity for spin control by the inquisitor. Am I just paranoid? Or is that realistic?

Rys, I was only referring to the spiking CPU load whenever someone accesses the Python page or our TRAC server... It's a good thing we don't have too many concurrent accesses, or response time would slooow down. Still, it's very useful, so I'm not complaining :)

From all the scripting languages (or shell replacements) I've seen, Ruby is by far the coolest, even if it's slow. I think if you did something like Unix's "init" in it, it would in fact be *faster*, because you could use muliple threads when dependencies allow (ok, shell allows that too, but shell is awkward). Running sequential shell scripts at bootup and waiting for disk isn't really high tech...

Heretix is a Linux distribution which uses Ruby for package management & init scripts. Don't know how multi-threaded the init script is though ..

What really is cute is how the Ruby yield statement makes writing iterators so very very very easy compared to say writing a C++ iterator, both conceptually and in Lines of Code.

Any cuter than in Icon?

It is ages and ages since I last looked at Icon. Is it still around?

Ask, google....yup. There it is...pull down pdf book, yield, hmm, yield, yield,...

Nope, don't see a reference to yield. Do you have a reference to it directly?

Just looking at Icon generally I note (and remember) it's main "thing" is generators.

Now that's interesting. I have found writing a generator for a arbitary sequence (in C/C++/Fortran/Basic/Pascal/.....) often quite challenging. Somehow everything seems to get spread over several methods and you need to store state and then basically somehow hop out of the inner loops and (conceptually) hop back into the middle of it again.

With Ruby it turns out easy and hard again.

Easy in the sense that you can develope like this...

# I want to loop over all doodads...

someSetUpCode
while( haventGot to the end)
   print doodad
   find next doodad
end

Really easy, straight forward you don't have to think.

The trouble is that is not easy to package up and serve to a client code. You can do something like...

# I want to loop over all doodads...

def all_doodads
  array = []
  someSetUpCode
  while( haventGot to the end)
     array << doodad
     find next doodad
  end

  return array
end

array = all_doodads
array.each do |doodad|
  doSomethingWith(doodad)
end

But that is yucky since you first have to generate all doodads and find storage for them before you client can start using them.

Ruby makes the evolution and refactoring of this sort of code really really easy with yield....

# I want to loop over all doodads...

def each_doodads
  someSetUpCode
  while( haventGot to the end)
     yield doodad
     find next doodad
  end
end

each_doodad do |doodad|
  doSomethingWith(doodad)
end

Really easy really straight forward.

The trouble with that is it forces you to chew on _every_ doodad. Not really a problem since you can skip them or even break out if you want...

Her is one of my favourite idioms.

require 'find'

def find_file_that_contains( string)
  Find.find( "/home/johnc/work") do |path|
    Find.prune if path =~ %r{/CVS $ }x # Prune out CVS meta directories.
    next if path =~ %r{ (\.bak | ~) $ }x Skip backup files
    next unless FileTest.file? path
    return path if File.read( path) =~ %r{#{string}}
  end
end

puts find_file_that_contains( 'This string')

But the point about generators is sometimes you want to suck on the teat once, sometimes twice and sometimes in different places.

So for some tasks, relatively rare, but certainly exist, generators are GASP SHLOCK HORROR just plain nicer than rubies "each/yield" mechanism.

No problem. Ruby always has had a generator module that implements generators via call-cc. Really hairy coding in there, but it works.

Wrap up _any_ object that has an "each" method in a Generator and away you go.

Well, still a small problem.

It's slow. Rubies old generator module was just plain slow slow slow treacly.

Then someone, I don't know who, got very smart. Instead of doing call-cc they put in an extra thread in the generator object, it generators objects as needed and packed them in a queue... Basically a halfway point between the one on one call-cc slow implementation and the resource hungry generate everything and store it in an array implementation. (Hint, Click on method names in rdoc to see implementation)

It is Way way faster. So Ruby has both very very easy for the client and easy for the service each / yield paradigm AND for those gnarly "suck on a teat" cases, fast thread based generators where the service code is written in the same easy / neat using yield fashion.

Nope, don't see a reference to yield. Do you have a reference to it directly?

"suspend" is the Icon term.

suspend === yield pretty much exactly as far as I can see.

Thus the most significant difference I can see between Icon / Ruby is this "Goal Directed Evaluation with Backtracking" thing in Icon. Ruby does not have that natively.

I suspect that would permit some more concise code in Icon than in Ruby. But I also suspect it may be very easy to write "write only code" using that stuff too. (Answer: Don't do that then!)

I'm also not entirely convinced it needs to be a language level feature, but that could easily be ignorance on my part.