Lambda the Ultimate - Python

Skipping C - SPE and synthetic programming in Python

Tue, 14 Aug 2007 04:11:16 -0400

Expression and Loop Libraries for High-Performance Code Synthesis. Christopher Mueller and Andrew Lumsdaine. LCPC2006.

To simultaneously provide rapid application development and high performance, developers of scientific and multimedia applications often mix languages, using scripting languages to glue together high-performance components written in compiled languages. While this can be a useful development strategy, it distances application developers from the optimization process while adding complexity to the development tool chain. Recently, we introduced the Synthetic Programming Environment (SPE) for Python, a library for generating and executing machine instructions at run-time.

The authors didn't show much interest yet in supporting the most widespread ISAs, those for Intel processors. Instead they focus on PowerPC but also Cell. Have fun with Python and PS3 hacking.

Edit: You might also checkout the site of Christopher Mueller containing related material.

Google Python Sprint 2007

Mon, 13 Aug 2007 17:37:28 -0400

Joining this event (August 22-25) or following the action from afar may be a good way to keep up with Python 3000.

The first alpha release (3.0a1) should be just around the corner.

Python 3000 Status Update

Wed, 20 Jun 2007 03:19:48 -0400

Guido updates the vision. Syntax focus seems cancerous, leaving FP sandwiched between set literals and backtick syntax, under Miscellany:

reduce() is gone. This doesn't mean I don't like higher-order functions; it simply reflects that almost all code that uses reduce() becomes more readable when rewritten using a plain old for-loop. (Example.)

lambda, however, lives.

Guido says that example "gives reduce() a bad name" which leaves me wondering as to its relevance. The idea that FP = Miscellany makes me wonder why the more stunning Python success stories (Google) involve FP techniques. Elsewhere:

zip(), map(), filter() return iterables

So FP is not even a free-standing category yet. I sometimes wish PLT Spy would revive, or that some FP language would target Python intermediate code. The value of Python is often stated to be its libraries.

I also wonder if Python people might build intuition by watching REBOL (=Lisp) people at work. They seem to enjoy puzzles. The motivating notion would be "hey! that REBOL trick should be so easy in Python."

Python in Pardus Linux

Tue, 19 Jun 2007 18:55:33 -0400

Pardus Linux is a case study of functional Python. It's a Linux distribution built from semi-scratch, the main focii being package management and init subsystems - places where C and shell script make poor sense. A funded group has finally tackled these issues.

A package management software deals a lot with sets, lists, and dependency graphs....We have extensively used functional operators (map, filter, reduce) and list comprehensions, even metaclasses are used in a few places.

Someone nudge Guido. Scheme or Oz might have been the better choice, but give them credit. They admit frankly to social acceptance issues.

Charming Python: Decorators make magic easy

Fri, 05 Jan 2007 09:40:42 -0500

Python made metaprogramming possible, but each Python version has added slightly different -- and not quite compatible -- wrinkles to the way you accomplish metaprogramming tricks. Playing with first-class function objects has long been around, as have techniques for peaking and poking at magic attributes. With version 2.2, Python grew a custom metaclass mechanism that went a long way, but at the cost of melting users' brains. More recently, with version 2.4, Python has grown "decorators," which are the newest -- and by far the most user-friendly way, so far -- to perform most metaprogramming.

While metaprogramming is inherently a bit confusing, I think this article could have a been a little clearer. Still, it's a nice highlevel introduction to decorators.

Type inference for Python

Mon, 29 May 2006 21:54:44 -0400

The subject of type inference for dynamically-checked languages came up in the Buried Treasure thread. A question was raised in that thread having to do with why static type inference in these languages is difficult. Since there's a nascent body of literature which addresses that question, here are a few links to articles and papers about type inference for Python.

A nice overview can be found in Localized Type Inference of Atomic Types in Python, a Master's thesis by Brett Cannon. The whole thesis is relevant, but for an overview of the issues, see Chapter 3, "Challenges of Inferring Types in Python". Chapter 4 summarizes previous attempts involving static inference in Python, including Psyco (previously on LtU) and Starkiller. The limitations of these attempts are briefly addressed.

Type inference solutions for Python invariably involve restrictions to make the problem tractable. The above paper focuses on "inferring atomic types in the local namespace". Another approach is described in Aggressive Type Inference, by John Aycock. Aycock makes an important observation:

Giving people a dynamically-typed language does not mean that they write dynamically-typed programs.

The article offers a type inference approach which exploits this observation. (If the meaning of the above quote isn't clear, I recommend reviewing our mammoth three-part thread on the subject, "Why type systems are interesting", part I, part II, and part III.)

The PyPy implementation of Python in Python (previously on LtU) uses a restricted subset of Python, called RPython, to implement parts of the language. RPython is sufficiently static to be able to support full-program type inference. It is not a "soft" inference approach, and is not designed to be used with ordinary Python programs. The paper Compiling dynamic language implementations covers the approach used for static analysis of RPython. The PyPy Coding Guide, starting at section 1.4 may also be useful.

(It may be interesting to note that the PyPy approach is very similar to that used previously for Scheme 48. The core of Scheme 48 is implemented in PreScheme, a subset of Scheme that supports full-program type inference.)

Finally, Guido van Rossum has a number of blog entries on the subject of adding optional static typing to Python:

If anyone knows of any other good treatments of type inference in Python or similar languages, please post links here.

EasyExtend - Python MetaProgramming

Mon, 22 May 2006 18:37:55 -0400

Just saw this announcement on Google groups / comp.lang.python.

EasyExtend is a constructive approach to extend the Python language using pure Python. EasyExtend is developed as a Python framework depending only on tools provided by the CPython interpreter suite ( compiler ) and standard library as well as some pieces of code borrowed from the PyPy project. Opposite to toolkits for writing extension modules in C ( or RPython in future ) the EasyExtend framework is dedicated to extend the language itself by adding new grammar rules and transformation of parse trees. Acting directly on the nodes of syntax trees makes EasyExtend safe and extensible. Moreover the parser and the transformations are considerably fast. While EasyExtend can obviously be used to define "little languages" embedded in Python it can also be used to create Python oriented tools like a code coverage tool based on code generation. For both use-cases examples will be provided.

You'll want to probably want to check out the examples.

Python 2.5a1 released

Thu, 06 Apr 2006 01:42:15 -0400

Python 2.5 seems to be feature complete now and is released as a first alpha. See here for a complete list of new features.

From a language perspective enhanced generators and the new with-statement are probably the most interesting features. For many developers the incorporation of the small relational database sqlite, the new XML package elementree and the foreign function interface ctypes might be the highlights.

Class decorators in Python

Thu, 30 Mar 2006 10:48:05 -0500

Guido resisted the few calling for class decorators, because there wasn't a clear use case that wasn't more readable done another way... [but] Guido has conceded, class decorators will make it into some future version of Python.

More + links: here.

Tail call elimination decorator in Python

Tue, 28 Feb 2006 02:33:24 -0500

Features of a programming language, whether syntactic or semantic, are all part of the language's user interface. And a user interface can handle only so much complexity or it becomes unusable. This is also the reason why Python will never have continuations, and even why I'm uninterested in optimizing tail recursion.

Thus spoke Guido - as LtU readers already know.

Now, not even four weeks later, it has become clear that turning tail recursions into iterations can be achieved by an innocent little decorator in pure Python. No Rube Goldberg machine(s) in sight.

Guido: Language Design Is Not Just Solving Puzzles

Sat, 11 Feb 2006 12:33:11 -0500

And there's the rub: there's no way to make a Rube Goldberg language feature appear simple. Features of a programming language, whether syntactic or semantic, are all part of the language's user interface. And a user interface can handle only so much complexity or it becomes unusable.

The discussion is about multi-statement lambdas, but I don't want to discuss this specific issue. What's more interesting is the discussion of language as a user interface (an interface to what, you might ask), the underlying assumption that languages have character (e.g., Pythonicity), and the integrated view of semantics and syntax of language constructs when thinking about language usability.

PyPy

Sat, 28 Jan 2006 06:47:49 -0500

PyPy, the Python implementation written in Python, was mentioned here a couple of times in the past. After it was mentioned in a recent LtU discussion, I took another look, and boy did they make a lot of progress when I wasn't looking. PyPy can even compile itself now... You should check it out again if you are interested in this sort of thing.

There's even an introduction to the techniques used by PyPy, including a nice (but very high level) overview of abstract interpretation.

MyHDL

Thu, 26 Jan 2006 11:53:12 -0500

(via Daily Python-URL)

The key idea behind MyHDL is the use of Python generators to model hardware concurrency. Generators are best described as resumable functions. In MyHDL, generators are used in a specific way so that they become similar to always blocks in Verilog or processes in VHDL.
A hardware module is modeled as a function that returns any number of generators. This approach makes it straightforward to support features such as arbitrary hierarchy, named port association, arrays of instances, and conditional instantiation.

MyHDL is an open-source package for using Python as a hardware description and verification language. A Verilog converter is also included.

EE Times provides some background on MyHDL in this article.

NetKernel - XML processing pipeline

Mon, 26 Sep 2005 14:12:59 -0400

It rapidly became clear that a single language runtime is too limited for general applications ... as a minimum we needed both a linear-flow language and a recursive tree composition language ... while declarative languages are excellent for rapid assembly of XML operations, they are terrible for expressing business logic and logical flow-control ... Our other declarative language is XML Recursion Language (XRL). XRL is like XInclude with services, in which inclusion references fire service invocations into the URI address space in order to recursively compose an XML document. XRL is an elegant and powerful way of building XHTML applications ... The active URI, in combination with the local NetKernel environment, is a functional program - Introducing NetKernel.

Main site; Tour.

It's another XML pipeline (there's a Freshmeat project that lets Coccoon apps run in NetKernel), apparently from HP, which might interest people here.

Guido van Rossum: Building an Open Source Project and Community

Mon, 05 Sep 2005 08:04:43 -0400

A long (close to two hours) audio presentation about Python's history and philosophy.

LtU readers will find a lot to disagree with (especially in part 2), for instance when Guido discusses dynamic languages and typing. And yet, I think Guido tries to be reasonable, even though I disagree with some of his conclusions.

Be that as it may, this talk provides a useful summary of Python's history, and some idea about the workings of the Python community. Long time readers will remember that I think language communities play an important, yet under appreciated, role in language evolution and success.