Lambda the Ultimate

Octave and Sage are different animals. Sage's goal is to be a replacement for Maple, Mathematica, and Matlab. It provides numerical tools (largely via SciPy), but also a wide range of tools for symbolic math. Octave's goal is somewhat narrower, being primarily to provide a Matlab-compatible numerical computation system. There's apparently a Sage interface to Octave (although I haven't tried it), in case you want to use Octave to do numerical stuff from Sage instead of using Sage's built-in numerical libraries.

[Edit: "replacement" probably wasn't the best term to use. "Substitute" might have been better. Or some other word that indicates similar functionality but not syntactic compatibility.]

Replacing Mathematica with something freely available seems more of challenge. Any good alternatives (hopefully that can read Mathematica files)?

My understanding is that Sage strives to replace the aforementioned systems in the sense of offering comparable functionality, not a replacement in the sense of GNU Octave, which strives to be more-or-less a drop-in replacement for Matlab.

And, Sage tries to offer a uniform interface to Mathematica and Maple and Maxima so that these systems can be more easily compared. I don't know how big the uniform interface is, and how often you need to fall back to a CAS-specific functionality.

In many fields (e.g. machine learning), Matlab is a sort of lingua franca. Octave's major advantage is that it can run a colleague's Matlab code with almost no changes. You should really ask yourself how many people are looking for yet another numerical programming language.

You should really ask yourself how many people are looking for yet another numerical programming language.

I'm not entirely certain why you addressed that question to me. I'm not a Sage developer, nor even really a Sage user (more of an occassional dabbler). However, since you've asked: anecdotally, I've observed that SciPy is becoming increasingly popular (both in academia and in industry), so the answer to your question may be "more than you think". That said, I'm personally a heavier user of Matlab than SciPy at the moment. Although I go back and forth between them (and sometimes Octave) depending on what I need to get done, Matlab clearly has a richer set of domain-specific toolboxes than SciPy at this stage. That obviously makes it more attractive for certain applications.

All of that's rather beside the point though. As I mentioned in my earlier comment, Sage isn't intended to be a direct drop-in replacement for Matlab. It's not even intended to be a numerical programming language. Sage is intended to provide a common Python-based interface to a variety of existing open-source (and some commercial) mathematical tools. Sage incorporates packages to tackle problems in (quoting from the Sage website)

...algebra, calculus, elementary to very advanced number theory, cryptography, numerical computation, commutative algebra, group theory, combinatorics, graph theory, exact linear algebra and much more.

I have to admit I am still not entirely sure I understand the model... Is the idea to have a single language (essentially Python), which uses the APIs of all these packages under the hood, or merely to allow users to embed calls written in Mathematica, Matlab etc? I figure it's the former, which would mean that as a language Sage does directly compete (or has to keep up) with all these languages.

I believe that the idea is to have a single "language" that uses package APIs under the hood (although that hasn't been fully implemented yet, so there are situations where you end up making an embedded call to the underlying package). My response to Sean was focused on his question regarding the need for a new numerical programming language - the point being that Sage isn't intended to be "just" a numerical programming language (let alone a Matlab-compatible one). That's apparent just from the name, as reported in the article by Stein that Leon started this thread with:

SAGE = Software for Algebra and Geometry Experimentation

I'm not entirely certain why you addressed that question to me.

I meant it for the broader "community" of people looking at yet more dialects for numerical computation. From my understanding of your comment, Sage is intended to make "Python + some libraries" a preferred environment for numerical programming. This effort seems pointless: each community's algorithmic knowledge is far more important than some syntax. I hate Matlab's OO and string handling, but I accept both when I want to use someone's numerical code.

I don't use matlab as it is not aimed at the areas that I work in, so my knowledge of it is entirely second-hand. But I do hear from people that its language is pretty awful, and a pain to work in (or around). I have played with Sage a lot, and it is a very nice environment to work in. Python is a nice language for hacking things up in, the APIs are generally quite sensible, and the notebook interface is very cool for exploratory work.

Having said that; for the domains that I work in I tended to use Magma, Maple, Graphviz, Gnuplot and lots and lots of glue. So in some sense I guess I am the target audience for Sage. And I haven't picked up either Magma or Maple in over a year. Working in Sage is definitely a good replacement for them. Matplotlib has won me over from gnuplot, and the sense of integration at having everything in one language means that Sage is far from pointless. I still write glue, but less of it, and everything being glued, and the glue itself, is in Python which makes it easier.

I don't understand your claim of pointlessness. If I accept your argument that the important thing in each community is algorithmic knowledge then you still seem to be assuming that old code is more important than new code. As a researcher I don't mind losing access to code in my domain - as you say the knowledge to recreate it is more important - but I do value an increase in productivity in writing new code. Certainly it is a different trade-off for everyone, but I spend more time working with new code, than working with old code. I can see the point in a project that makes that task easier.

In my experience, the real bottleneck is that most statistical analysis is provided to scientists as "packages" that all scientists share. I can't remember who made the quip, I think it was Richard Feynman, but it goes something like "Scientists of the 19th century shared each others research methods; scientists of the 20th century shared each others FORTRAN bugs". Substitute C++ (via Fermilab/CERN Scientific Linux environment targeted towards physicists) for FORTRAN, and you've got what todays supercolliders experiments statistical analysis are built on.

There was a really nasty, but academic standard MATLAB package I had to use when doing human brain image analysis. I had to modify some of its internal subsystems so that the proper hooks were provided for what I wanted to do (I spent days just reading the code before I made any changes, for fear of introducing a bug). This project was done by some neurology major, and it morphed into a huge academic project with many contributors.

the real bottleneck is that most statistical analysis is provided to scientists as "packages" that all scientists share.

This is a *feature*. The idea is to create a single robust, reliable version of things that are tricky to get right (e.g. because of numerical stability), then encourage everyone to use it. I would rather have everyone share the same small set of FORTRAN bugs than have everyone fight with his own larger set of FORTRAN/C++/sage bugs.

It can be both a feature and a bottleneck...

Allan is right, Sage's goal is more ambitious than "just" replacing Matlab. Here's a pretty impressive list of the components of a Sage distribution.

New but promising, Incanter is Clojure plus math/statistical/graphing libraries: http://incanter.org/

Some of the early contributors are interested in using this on top of the scaling infrastructure that is popular in the Java world, such as Hadoop and Cascading.

Wow! This sounds very exciting.

Incanter is pretty cool, but I'm a bit concerned and puzzled by it, in that Lisp-Stat failed pretty miserably. By "miserably", I don't mean that it was miserable, just that it had a lot of promise and never got anywhere.

I sort of have this impression that the statistical community has already sort of voted on their preferences for a Lisp-like statistical language. Those who experienced that period and were exposed to Lisp-Stat and R and watched R rise so prominently while Lisp-Stat was abandoned will probably be really reluctant to adopt a modern clone.

Then again, maybe R's popularity will highlight its shortcomings and lead to widespread interest in Incanter.

Back to the Future: Lisp as a Base for a Statistical Computing System

Essentially, the latest developments in statistical languages are that the people who founded R/SPlus now think that a Lisp-like language might be the best direction for the statistical language community.

I don't know if this really addresses your objections.

Another counter-point would be in physics research. Per Bruun recommended physicists stop coding in Fortran and move to C++, during a time period when physicists were increasingly frustrated with standard Fortran environments poor support for dynamic activation records and other neat programming language supportable tricks. (To do dynamic memory tricks, they would often use buggy third party software like Winteractive that would leak memory and was very unstable.) A lot of physicists at Fermilab and CERN dismissed Per as an old timer who didn't know what he was talking about. So he quit the committee and went on his own, and beat his dissenters back with a working implementation that addressed physicists needs while the other project stalled and never really got off the ground due to effective bike shed arguments.

....apologies if this history is a little off. Parts of the story were supplied to me by a dude who manages a supercollider, and are thus second-hand.

I guess that paper deserves its own thread (hint, hint)...

Back to the Future: Lisp as a Base for a Statistical Computing System. and the accompanying slides.

I honestly thought it was already posted to LtU before, and right before I added it I search the archives and didn't find it.

I'm aware of that paper by Ihaka and Lang. It's very interesting in a number of ways, and deserves more widespread attention.

I guess my reaction to that paper, reinforced shortly afterwards with Incanter, was that many of these issues were being discussed by Tierney awhile ago with Lisp-Stat. I thought Lisp-Stat had a lot going for it, and while it gained some traction, it lost it really readily.

So now we have Incanter, and I'm really cautious about it, because I feel like I've been there before. I don't know if people were turned off by a Lisp-style syntax in the statistical community, or really liked R, or both, but I feel like it's worth revisiting why Lisp-Stat didn't gain traction before pushing a statistical Lisp all over again, if for no reason other than to avoid the same mistakes. It seems all the more important given that (1) other viable options like Haskell, Python, or Scala (or maybe even Fortress someday?) might fill some of those gaps (whatever they might be), and (2) R is even more popular than it was during Lisp-Stat's peak.

My informal observations indicate that scientists will use the tool that is (a) most widely used around them and (b) easiest to get their hands on. I guess that's not a revelation to anyone... As a consequence different cliques/disciplines use different tools. Most of these tools are in the Matlab genre rather than the Haskell genre (if you get my drift). The more DSLs are used, the better chance there is to retrofit types systems and other support tools. This is one reason why the adoption of R is encouraging. Other areas, based on anecdotal evidence, are doing worse.

I agree, most will use the tools they find ready to hand, around them, taught in their classes, etc.

But it's important that some fraction use tools that are actually more useful, even if less common.

A couple of years ago (it seems) I mentioned that at my company a very uncommon programming language was adopted for design. This was "MainSAIL" (or maybe "MainSail"), a language used for CAD of ICs that was based on the SAIL language out of the Stanford AI Lab, ergo the name.

Obscure, yes. But it probably gave my company a few years' lead over companies still struggling with FORTRAN or even ALGOL or (the still primitive) Pascal or the (dead on arrival) PL-1.

Intel used it from the 70s through the 80s. (I wouldn't be surprised if it wasn't still being used in pockets into the 90s.)

I think some recent examples where companies used Python or Ruby for "social apps" are also examples of this.

Not sure of the PLT connection, so I'll stop here. But it says that even obscure languages have places to shine, sometimes not clear to the outside world.

For the record, I think that in many cases their choice of tools is a reasonable pragmatic decision, even if personally I prefer cooler packages.

Edited to add: The following quote from the article is an example of issues I had in mind: I then realized that if I did this, I would have to do it by myself, since almost everybody I knew used Magma, and would consider my plan too difficult and pointless. I wouldn’t get to do number theory for years. My spirit broke.

I leave it open how this applies to current efforts.

By "language design" Cannon meant the core programming language, including features such as "user defined types", "exception handling", etc. Cannon's remark is relevant to Sage in at least two ways:

1. For the Sage project, for the most part language design in the above sense is done via the Python community. Because Python is a general purpose widely used language, it receives strong financial support from industry, including Google, which heavily uses Python. If Sage developers want to do language work that impacts Sage, they do it through improving Python (or Cython). This means that the funding constraints Cannon had for Magma do not apply, since the range of funding sources for Python are much broader. (Sage does have a minimal preparser, but that's trivial in comparison to the serious language design issues like I was discussing with Cannon.)
2. The Sage project is currently organized in such a way that it is not fundamentally dependent on grant funding. Most people (and there are about 200) who work on Sage receive no financial compensation for their efforts -- they work on Sage because they very strongly believe in the value of the project and find it personally useful for their work. For example, our current amazing release manager, Mike Hansen, has been working on Sage for the last 6 months as a volunteer, and our main technical editor, Minh Nguyen, is an undergraduate at the University of Melbourne who is also currently not paid to work on Sage. As a community we can do whatever we feel should be done, without being beholden to funding agencies (or paying customers). That said, we love to get user feedback, and we have received financial support (from Google, Sun, NSF, Microsoft, DoD, etc.), which we're very, very grateful for--it just isn't something we've come to depend on.

Welcome to LtU!

Thanks for the insider view on Sage and funding. These are interesting and important points. My intent was to focus on those cases that do require language engineering, that is the cases in which the features of the general purpose language (if the domain specific language is embedded, as in Sage) are not enough. This may be relevant to Sage (as I mentioned above, I am unsure about the semantic model), if you need domain specific extensions to core python semantics. My comment was meant to hint to other projects as well, in which a domain specific language, embedded or otherwise, may require language design work to evolve (e.g., R, and the discussion here).