Sunday, May 31, 2009

Dynamic Languages at TechEd 2009

TechEd 2009 (Microsoft developer conference) is come and gone, but dynamic languages played a bigger part there than ever before and John Lam (IronRuby commander in chief) has blogged about his presentation.

He has been working on interesting twists on the REPL; the Read-Eval-Print-Loop otherwise known as the interactive interpreter. He demonstrates one that can switch between DLR based languages (IronPython and IronRuby) and is an editor as well.

The story for dynamic languages in Visual Studio is not massively impressive, but you can get good results with Wing or Netbeans, SharpDevelop has excellent IronPython support and there is also the editor built into the very interesting Pyjama Project.

With the Microsoft dynamic languages team getting interested in editors perhaps the future is brighter for a supported Microsoft IDE for IronPython and IronRuby?
Dynamic languages on .NET are picking up momentum at this year's TechEd. Your typical TechEd attendee is a mainstream .NET developer, since this conference focuses on technologies that are shipping today (as opposed to futures conferences like the PDC). To speak more to this crowd, I focused on how they can make their existing .NET apps better by mixing in some end-user scripting.

My talk was DTL332 if you're a TechEd attendee (not sure when/if the videos will open up to the general public). I showed how you can add Ruby and Python scripting to an existing app, and spent some time building some simple REPLs. Toward the end of the talk, I showed a more realistic scenario where I embed a REPL in an existing Open Source .NET application: Witty.

During the demo, I showed off a twist on the traditional REPL - one where the editor and the REPL are one and the same. To run code in this REPL, you select the code that you want to run, and the output appears immediately below the selection. Since the REPL is just an editor, you can just delete any output that you don't want. History behaves just fine as well.

Saturday, May 30, 2009

Two Englishmen Named Tim Review IronPython in Action

In the last couple of weeks two Englishmen called Tim have posted reviews of IronPython in Action.

Tim Anderson is a technical writer who specialises in writing about Microsoft technologies. As well as writing for the Guardian he blogs regularly and has just reviewed IronPython in Action:
It is aimed at Python programmers who do not know .NET, and .NET programmers who do not know Python, rather than at existing IronPython developers. The authors run through the basics of Python and its .NET integration, so that by the end of Part 1 you could write a simple Windows Forms application. Part 2 is on core development techniques and covers duck typing, model-view-controller basics, handling XML, unit and functional testing, and metaprogramming – this is where you generate and execute code at runtime.

It’s a well-written book and fulfils its purpose nicely. I like the way the book is honest about areas where IronPython is more verbose or awkward than C# or VB.NET As someone who knows .NET well, but Python little, I could have done without all the introductory .NET explanations, but I understand why they are there. I particularly liked part two, on core programming techniques, which is a thought-provoking read for any developer with an interest in dynamic languages.
Tim Golden is a Windows Python guru. He is the author of the Python WMI module which interfaces Python to Windows Management Instrumentation. Tim's help was invaluable in writing the WMI section of the system administration chapter of IronPython in Action. This earns him a mention or two in the footnotes, a fact Tim notes in his review...
By way of disclosure, I’m given a couple of blushingly generous footnote credits by Michael which naturally leave me feeling well-disposed towards the book as a whole. But even without those, I’d be giving it the thumbs-up.

But the most important win, I think, is managing to write a book about IronPython, not about Python or .NET. Naturally there is an element of explanation involved in both directions when some feature is being introduced or compared. But for the most part you can refer to the appendices which give summaries of Python/.NET if an unfamiliar term arises. For me, this achievement is key to the success of a book like this.

Friday, May 29, 2009

A Good Mix 7: JScript, Silverlight, Pyke and more

Another selection of IronPython and DLR related postings culled from the blogosphere in the last few weeks.
Managed JScript is an implementation of Javascript built on the Dynamic Language Runtime that shipped with Silverlight. It was touted as being useful for porting Javascript client-side code to work with Silverlight. It was only ever released in binary form and hasn't been maintained. In this blog entry the Frugal Coder laments the situation:
I'm a Developer, I build Web Applications, and have for well over a decade now, and I love Javascript. I find the DLR compelling, and really want to see the Javascript implementation treated like a first class citizen. Give me code, samples and howtos. I'm going down this path anyhow, and would rather not do it alone.
Pyke is a very interesting looking Python project: "Pyke introduces a form of Logic Programming (inspired by Prolog) to the Python community by providing a knowledge-based inference engine (expert system) written in 100% Python."

This blog entry goes through what it takes to get Pyke running on IronPython; running the towers of hanoi example.
Python is fantastic for scripting and system administration tasks, but in the .NET world Powershell got there first. This blog entry has an interesting take on which you should use:
Were it not for the fact that Powershell is as intuitive as Linear B, it would be game over. As it is, IronPython and IronRuby remain interesting alternatives. They make the scripting process far simpler, and the syntax is much closer to the type of programming language that DBAs are used to. They become even more attractive when one considers the versatility of the .NET Dynamic Language Runtime (DLR), which allows you to plug IronPython or IronRuby into your .NET applications, call C# methods and so on. Maybe, with these DLR-based languages, we really are moving closer to achieving the dream of all DBAs: one scripting language to do everything. However, your Sysamin will loathe them as he will be engrossed in the extended culture-shock of having to redo all his favourite scripts in Powershell.
An IronPython presentation by Guiseppe Turitto:
Last night (March 19th), I had the chance to give a speech about IronPython at the .NET users group New York. First thing, I have to say Thank You, to the .NET UG-NY leader’s, for the opportunity, and thank you to my fellows at the .NET UG-NY for attending.

Anyway for all of those that ask me for a copy of the slides, you can get them here together with the source code for the demos.
The obligatory non-English entry, this is a simple "Hello World" example using IronPython and XAML in Silverlight. It is great fun to program the browser in Python through Silverlight. I'm not a big fan of XAML (an XML dialect for describing user interfaces) but its ok if someone else does it... (Anything that can be done from XAML can be done from code - but there are occasions when the XAML is actually less verbose.)

IronPython and the elmcity project: Together again

Jon Udell has been using Azure (Microsoft's cloud platform) for a calendar project he calls elmcity. Initially he tried to use IronPython, but the security settings of Azure (medium trust only) made this impossible. He proceeded with the project in C#, but Azure can now be used with a full-trust environment so IronPython is back on the table...
In the first installment of this elmcity+azure series my plan was to build an Azure-based calendar aggregator using IronPython. That turned out not to be possible at the time, because IronPython couldn’t run at full strength in Azure’s medium-trust environment. So I switched to C#, and have spent the past few months working in that language.

It’s been a long while since I’ve worked intensively in a compiled and statically-typed language. But I love being contrarian. At a time when low ceremony languages are surging in popularity, I’m revisiting the realm of high ceremony. It’s been an enjoyable challenge, I’ve gotten good results, and it’s given me a chance to reflect in a more balanced way on the “ceremony vs. essence” dialogue.

Meanwhile, Azure has moved forward. It now provides a full-trust environment. That means you can run PHP, which is interesting to a lot of folks, but it also means you can run IronPython, which is interesting to me.

In this entry I’ll show you how I’m starting to integrate IronPython in the two main components of my Azure project: the web role that provides the (currently minimal) user interface, and the worker role that does calendar aggregation.

Thursday, May 28, 2009

Python on the .NET Framework: There's more than one way to do it

IronPython is written in C# and is an implementation of the Python programming language that runs natively on the .NET framework with full access to .NET libraries. It isn't however the only way of using Python with .NET. One disadvantage of IronPython is that unless you are using Ironclad you can't use Python C extensions with IronPython,

One alternative is Python.NET. This is a version of CPython (the normal implementation of Python) which also has integration with the .NET framework. As you would expect you can use Python C extensions with Python.NET, but when you use .NET objects you aren't using them natively but are using a proxied or copied version instead.

There is also a little known third option: PyPy.NET. PyPy is an extremely interesting project - it is both an interpreter compiler toolchain, allowing you to write interpreters for dynamic languages in a high level static language called RPython which is a subset of Python, and a Python interpreter written in RPython.

The compiler toolchain supports compiling for a variety of backends - compiling interpreters to C (the native backend), Java byte-code and also IL (Intermediate Language the .NET bytecode). This means that a single implementation (sourcecode in RPython) can be maintained for all three platforms.

A key feature of PyPy is that the toolchain can also generate and compile specialised JIT compilers for all three backends.

The PyPy team recently blogged about two papers that have been accepted for the ICOOOLPS (Implementation, Compilation, Optimization of Object-Oriented Languages, Programs and Systems) conference. One of these is on the PyPy JIT for .NET:
The relevant paper is "Faster than C#: efficient implementation of dynamic languages on .NET" (PDF) by Armin, Anto and Davide Ancona, who is Anto's Ph.D. advisor.
"Faster than C#" is a deliberately misleading quote. It refers to PyPy's claim (still unproven except for specific benchmarks) that PyPy can be "Faster than C". Faster than C means faster than the C implementation of Python but also alludes to the PyPy team's conviction that dynamic languages need not be slower than statically typed languages like C...

.NET already has a JIT, so it may seem odd for a .NET implementation of Python to also have a JIT. What it permits is for the language to emit efficient and typed bytecode that the .NET JIT is better able to optimise as machine code - a genuine double JITting! The paper does claim to demonstrate that with their approach code written in a dynamic language can run as fast, or faster, than the equivalent code in C#!

The paper abstract describes this:
The Common Language Infrastructure (CLI) is a virtual machine expressly designed for implementing statically typed languages as C#, therefore programs written in dynamically typed languages are typically much slower than C# when executed on .NET.

Recent developments show that Just In Time (JIT) compilers can exploit runtime type information to generate quite efficient code. Unfortunately, writing a JIT compiler is far from being simple.

In this paper we report our positive experience with automatic generation of JIT compilers as supported by the PyPy infrastructure, by focusing on JIT compilation for .NET.

Following this approach, we have in fact added a second layer of JIT compilation, by allowing dynamic generation of more efficient .NET bytecode, which in turn can be compiled to machine code by the .NET JIT compiler.

The main and novel contribution of this paper is to show that this two-layers JIT technique is effective, since programs written in dynamic languages can run on .NET as fast as (and in some cases even faster than) the equivalent C# programs.

The practicality of the approach is demonstrated by showing some promising experiments done with benchmarks written in a simple dynamic language.
Interestingly both Python.NET and PyPy.NET have copied IronPython in makingthe clr module the first point of integration with the Common Language Runtime and use the same syntax as IronPython for creating typed arrays and working with generics. This means that despite their differences code which uses .NET classes and features is 'somewhat portable' between all three implmentations.

Wednesday, May 27, 2009

Using IronPython with Wing and Netbeans IDEs

Any good Python IDE will work well as an IronPython IDE, but some specific integration helps. One very good option is SharpDevelop, which has recently gained some very sophisticated IronPython support.

My favourite Python IDE is Wing, particularly for its autocomplete (intellisense) for Python and how easy it is to extend with new tools through its scripting API. Unfortunately Wing doesn't know about .NET types and so doesn't provide autocomplete when using them through IronPython. It does have a mechanism for teaching it about new libraries (generating interface files) which I modified to work with .NET libraries.
This article has the PI (Python Interface) script for downloads with instructions on how to generate PI files for .NET assemblies / namespaces plus how to make them available for Wing. It also has an example extension script for Wing which launches the current file with the IronPython interpreter (with instructions on how to make it available to Wing and bind it to a key combination).
I'm not the only one who has been experimenting with using IronPython with Python editors. Steve Gilham has been using Netbeans (the Java IDE by Sun which recently gained Python support) with IronPython:
With the arrival of the IronPython 2.6 beta 1, with ctypes support, the IronPython 1.x only support in IronPython Studio really takes the IDE from a bit dated to seriously obsolescent.

Now, I had tried to get IronPython 2.0 to talk with NetBeans, but hadn't immediately achieved success, so, with less motivation, had put that to one side. Perhaps for similar reasons, a recipe for this wasn't out on Google already -- but now I really needed to crack the problem, I wasn't going to let that earlier abject failure put me off...

Two New Releases: DLR 0.9.1 and IronPython 2.6 Beta ASP.NET Integration

To go with the four releases of IronPython and IronRuby are two more new releases:
A new source release of the DLR. This includes the Sympl sample language and can be built with Visual Studio 2008:
This release has several design cleanups and final calls for the bits we’re shipping in CLR 4.0. This release is also consistent with Visual Studio’s Beta1 release in that these sources (with namespace tweaks) were used for parts of CLR 4.0. Documents have all been updated, and there is a new walkthrough document for an example language implementation on the DLR. The language, Sympl, has a C# and an IronPython implementation in the source tree, along with some example code written in Sympl. There are many one-off examples for Expression Trees node types that are in the DLR source tree now.
This is an updated release for using IronPython with ASP.NET and is compatible with the recent IronPython 2.6 Beta 1 release. Amongst other things it fixes the problem of using IronPython with ASP.NET and .NET 3.5 which were present in the last release. For this reason it was a highly anticipated release. Part of the delay has been finding a Microsoft team to take responsibility for the integration. It was 'owned' by the ASP.NET team who showed little interest in maintaining it. The hope is to get authorization for a source release and including it in the main IronPython project. In the meantime Jimmy Schementi did the work for this build.
This release is compatible with IronPython 2.6 Beta 1. Currently it does not include Language Services Support and project templates. To create a new IronPython ASP.NET WebForms project, simply copy the “examples\hello-webforms” to a new directory and rename the directory to your liking. A redistributed copy of IronPython 2.6 Beta 1 can be found in the “bin” directory; all files except Microsoft.Web.Scripting.dll, the IronPython ASP.NET integration, are from the IronPython 2.6 Beta 1 release.

Included in this release are two WebForms examples that are written in IronPython: “hello-webforms” and “album-handler”, which can be found in the “examples” directory. “hello-webforms” is a simple web application that shows PostBack handling, and “album-handler” is a larger web application that creates a photo album from a directory of images and generates thumbnails for them on the fly.

Saturday, May 23, 2009

Python, Jython and IronPython Performance

Robert Smallshire has been working on an implementation of BBC Basic written in IronPython.

During development he has been disappointed with the performance of IronPython, in fact he discovered that some of his code was running massively slower than the same code on either CPython (the native implementation of Python) or Jython (the implementation for the Java Virtual Machine).

He posted a series of blog entries on this where he dug deeper into the problem and eventually, with the help of the IronPython team he found a solution. I've listed the blog entries in the order they were posted; what follows is a story of performance, impressive charts and an explanation:
Significant claims have been made about the performance of IronPython, notably back at its inception in 2004 when Jim Hugunin, creator of both IronPython and its cousin Jython, presented a paper on IronPython performance at PyCon. Since then, there have been numerous claims to IronPython’s supremacy over CPython in the performance stakes. The IronPython Performance Report reiterates that IronPython can turn in a good performance. According to Hugunin the standard line we’ll see is,

“IronPython is fast – up to 1.8x faster than CPython on the standard pystone benchmark.

But do these claims stand up in the face of real-world Python code?

The claims of good performance are based on synthetic micro-benchmarks which don’t accurately reflect balance of coding techniques found in complete programs.

At this point I’d like to offer my own quote:

“IronPython can be slow – 10x to 100x slower than CPython on real-world code and it has been observed to be up to 6000x slower.
The blog entry features some of the aforementioned pretty charts illustrating the performance of the Owl Basic code on Jython, Python and IronPython. The IronPython code is running orders of magnitude slower. Jython fares remarkably well however.

Naturally this doesn't match my experience of working with IronPython, but the story continues...
In this entry Robert has found an isolated example that seems to demonstrate the performance problem; a simple test program which builds a simple binary tree of objects.

The charts show that for small depths CPython outperforms Jython, but for deeper object graphs Jython scales better than CPython. IronPython seems to perform astonishingly badly from start to finish!

In the comments Curt Hagenlocher (IronPython core developer) notes that changing the counter from updating a class variable to a global variable makes a slight difference to the performance of IronPython.
The code Robert Smallshire was using to benchmark was updating a class variable as a counter. This is perfectly valid Python code, even idiomatic code. Unfortunately it triggered a code path in the IronPython implementation that could be described at best as "unfortunate". More on that shortly.

When Robert updated his code following Curt's suggestion, the results were interesting:
This made a phenomenal difference with IronPython completing in only 12% of the time taken by CPython – over 8× faster with a binary tree depth of 20.
Dino Viehland (IronPython core developer) followed up with a blog post explaining the problem, and how they were fixing it in IronPython. The fix is now in place and will be in IronPython 2.6.
Ahh, performance… It’s so much fun! It’s easy to measure and as you work on it you can see immediate benefits (or not!). I personally enjoy getting to spend time looking at interesting performance issues and I’ll use this as a time to plug IronPython 2.6 Beta 1 where we’ve continued our focus in this area. Even more fun yet is when you compare and contrast different implementations - it can be both surprising and amusing. Recently Robert Smallshire has been doing just that. Obviously Robert has found a case in IronPython where something is performing horribly. I’d like to take a look at what’s happening here and what we can do to improve this case.

Ok, but let’s get onto the fun technical details! It all comes down to one line of code “Node.counter += 1”. That’s a pretty innocent looking piece of code, how can it cause so many problems? First you need to understand a little bit about how IronPython works. IronPython internally keeps versions of type objects and uses this information to generate optimal code paths for accessing those objects. So the next line of code in this example, “self._children = children” usually turns into something that’s like:

if (self != null && self.GetType() == typeof(PyObject) && self.PythonType.Version == 42) {

self._dict["_children"] = ;

}

What that one line of code causes then is for the version of the type to change and for this “rule” to be invalidated. So we’ll figure out what to do, generate a new cached rule, only to have it not work the next time around. The same thing applies to the calls to Node – we re-generate the “optimal” code again and again only to discover we have to throw it away.

...

So you might be curious - what’s the fix? Well in this case we’re just mutating the type by changing a simple value stored in the type (versus say adding a property or a method to the type). Today we get a very slight throughput performance benefit because we can burn the value directly into the generated code instead of fetching it from the wrapper the type object keeps it in. With the fix we’ll burn the wrapper in and fetch the value from the wrapper each time. Then when we go and modify the type we just modify the wrapper instead of the type and its version is unchanged! This fix is now checked into the IronPython source tree and you can download the fixed sources right now.

In conclusion I’d like to wrap up a little bit about our performance philosophy – performance is hard and deciding which numbers matter to you is important. We have tried to tune IronPython for apps which run for more than a fraction of a second. We have also tuned IronPython for code which is not constantly mutating types at runtime. And we’re always open to feedback when users encounter problems with IronPython and we’ll listen to what scenarios are important for them. But if you look at these graphs I think you can see that we’ve done a good job at making IronPython a fast implementation of Python.
In short, if you find code where IronPython performs dramatically worse than CPython - it's a bug! Report it on the IronPython mailing list; Dino, Curt and team are very responsive.

Friday, May 22, 2009

The Dynamic Keyword in C# 4.0

The new dynamic keyword in C# 4.0 is made possible by including the Dynamic Language Runtime in the .NET framework. The DLR was originally part of the first implementation of IronPython and was abstracted out in order to create a framework for implementing dynamic languages on .NET.

Further than that, the DLR has been used to introduce new features into C# and VB.NET, the static languages traditionally used to program .NET. This is seen as a major part of what is new in .NET 4.0.

When Jim Hugunin went to work at Microsoft he wasn't employed to work specifically on IronPython; he joined the CLR (Common Language Runtime) architecture team with the brief of helping to make .NET a better runtime for dynamic languages. What follows is largely as the result of his influence and work.

The full semantics of the dynamic keyword are laid out in the C# 4.0 language specification document.
Note that this document is for the beta version of .NET 4.0. Although the details are not likely to change much it will inevitably be revised and edited for the final release.
19. Introduction to C# 4.0
The major theme for C# 4.0 is dynamic programming. Increasingly, objects are “dynamic” in the sense that their structure and behavior is not captured by a static type, or at least not one that the compiler knows about when compiling your program. Some examples include
  • Objects from dynamic programming languages, such as Python or Ruby
  • COM objects accessed through IDispatch
  • Ordinary .NET types accessed through reflection
  • Objects with changing structure, such as HTML DOM objects
While C# remains a statically typed language, we aim to vastly improve the interaction with such objects.

Dynamic lookup allows you to write method, operator and indexer calls, property and field accesses, and even object invocations which bypass the normal static binding of C# and instead gets resolved dynamically.
20. Dynamic Binding
Dynamic binding provides a unified approach to selecting operations dynamically. With dynamic binding developer does not need to worry about whether a given object comes from e.g. COM, IronPython, the HTML DOM or reflection; operations can uniformly be applied to it and the runtime will determine what those operations mean for that particular object.

This affords enormous flexibility, and can greatly simplify the source code, but it does come with a significant drawback: Static typing is not maintained for these operations. A dynamic object is assumed at compile time to support any operation, and only at runtime will an error occur if it was not so.

C# 4.0 introduces a new static type called dynamic. When you have an object of type dynamic you can “do things to it” that are resolved only at runtime:
dynamic d = GetDynamicObject(…);
d.M(7);
C# allows you to call a method with any name and any arguments on d because it is of type dynamic. At runtime the actual object that d refers to will be examined to determine what it means to “call M with an int” on it.

The type dynamic can be thought of as a special version of the type object, which signals that the object can be used dynamically. It is easy to opt in or out of dynamic behavior: any object can be implicitly converted to dynamic, “suspending belief” until runtime. Conversely, the compiler allows implicit conversion of dynamic expressions to any type:
dynamic d = 7; // statically bound implicit conversion
int i = d; // dynamically bound implicit conversion
Not only method calls, but also field and property accesses, indexer and operator calls and even delegate invocations and constructors can be dispatched dynamically:
dynamic d = GetDynamicObject(…);
d.M(7); // calling methods
d.M(x: “Hello”); // passing arguments by name
d.f = d.P; // getting and setting fields and properties
d[“one”] = d[“two”]; // getting and setting through indexers
int i = d + 3; // calling operators
string s = d(5,7); // invoking as a delegate
C c = new C(d); // selecting constructors
The role of the C# compiler is simply to package up the necessary information about “what is being done to d”, so that the runtime can pick it up and determine what the exact meaning of it is, given an actual object referenced by d. Think of it as deferring part of the compiler’s job to runtime.

The result of most dynamic operations is itself of type dynamic. The exceptions are conversions and constructor invocations, both of which have a natural static type.

At runtime a dynamic operation is resolved according to the nature of its target object d: If d implements the special interface IDynamicObject, d itself is asked to perform the operation. Thus by implementing IDynamicObject a type can completely redefine the meaning of dynamic operations. This is used intensively by dynamic programming languages such as IronPython and IronRuby to implement their own dynamic object models. It can also be used by APIs, e.g. to allow direct access to the object’s dynamic properties using property syntax.

Otherwise d is treated a standard .NET object, and the operation will be resolved using reflection on its type and a C# “runtime binder” component which implements C#’s lookup and overload resolution semantics at runtime. The runtime binder is essentially a part of the C# compiler running as a runtime component to “finish the work” on dynamic operations that was left for it by the static compiler.

Assume the following code:
dynamic d1 = new Foo();
dynamic d2 = new Bar();
string s;
d1.M(s, d2, 3, null);
Because the receiver of the call to M is dynamic, the C# compiler does not try to resolve the meaning of the call. Instead it stashes away information for the runtime about the call. This information (often referred to as the “payload”) is essentially equivalent to:
“Perform an instance method call of M with the following arguments:
  • a string
  • a dynamic
  • a literal int 3
  • a literal object null”
At runtime, assume that the actual type Foo of d1 does not implement IDynamicObject. In this case the C# runtime binder picks up to finish the overload resolution job based on runtime type information, proceeding as follows:
  • Reflection is used to obtain the actual runtime types of the two objects, d1 and d2, that did not have a static type (or rather had the static type dynamic). The result is Foo for d1 and Bar for d2.
  • Method lookup and overload resolution is performed on the type Foo with the call M(string,Bar,3,null) using ordinary C# semantics.
  • If the method is found it is invoked; otherwise a runtime exception is thrown.
20.1 The dynamic type
The grammar is extended with the following type expression:
type:
...
dynamic
The types dynamic and object are considered the same, and are semantically equivalent in every way except the following two cases:
  • Expressions of type dynamic can cause dynamic binding to occur in specific situations
  • Type inference algorithms as described in §7.4 will prefer dynamic over object if both are candidates
This deep equivalence means for instance that:
  • There is an implicit identity conversion between object and dynamic
  • There is an implicit identity conversion between constructed types that differ only by dynamic versus object
  • Method signatures that differ only by dynamic versus object are considered the same
  • Like with object, there is an implicit conversion from every type (other than pointer types) to dynamic and an explicit conversion from dynamic to every such type.
The type dynamic does not have a separate runtime representation from object – for instance the expression
typeof(dynamic) == typeof(object)
is true.

An expression of the type dynamic is referred to as a dynamic expression.

20.2 Dynamic binding
The purpose of the dynamic type is to affect the way operations are selected by the compiler. The process of selecting which operation to apply based on the types of constituent expressions is referred to as binding.

The following operations in C# are selected based on some form of binding:
  • Member access: e.M
  • Method invocation: e.M(e1,…,en)
  • Delegate invocaton: e(e1,…,en)
  • Element access: e[e1,…,en]
  • Constructor calls: new C(e1,…,en)
  • Overloaded unary operators: +, -, !, ~, ++, --, true, false
  • Overloaded binary operators: +, -, *, /, %, &, &&, |, ||, ??, ^, <<, >>, ==,!=, >, <, >=, <=
  • Compound assignment operators: +=, -=, etc.
  • Implicit and explicit conversions
When dynamic expressions are not involved, C# always defaults to static binding, which means that the compile-time types of constituent expressions are used in the selection process. However, when one of the constituent expressions in the above listed operations is a dynamic expression, the operation is instead dynamically bound.

20.3 Compile time semantics of dynamic binding
An anonymous function cannot be used as a constituent value of a dynamically bound operation.

A method group can only be a constituent value of a dynamically bound operation if it is immediately invoked.

Other than that, a dynamically bound operation always succeeds at compile time, unless otherwise specified in the following.

The static result type of most dynamically bound operations is dynamic. The only exceptions are:
  • Conversions, which has the static type that is being converted to
  • Constructor invocations which have the static type that is being constructed
Most dynamically bound operations are classified as a value. The only exceptions are member accesses and element accesses, which are classified as variables. However, they can not be used as arguments to ref or out parameters.

20.3.1 Static binding with dynamic arguments
In certain cases if enough is known statically, the above operations will not lead to dynamic binding. This is the case for:
  • Element accesses where the static type of the receiver is an array type
  • Delegate invocations where the static type of the delegate is a delegate type
In these cases the operation is resolved statically, instead implicitly converting dynamic arguments to their required type. Thus, the result type of such operations is statically known.

20.3.2 Dynamic binding with a statically known candidate set
For most dynamically bound operations the set of possible candidates for resolution is unknown at compile time. In certain cases, however the candidate set is known:
  • Static method calls with dynamic arguments
  • Instance method calls where the static type of the receiver is not dynamic
  • Indexer calls where the static type of the receiver is not dynamic
  • Constructor calls
In these cases a limited compile time check is performed for each candidate to see if any of them could possibly apply at runtime. This check includes:
  • Checking that the candidate has the right name and arity
  • Performing a partial type inference to check that inferences do exist where not depending on arguments with the static type dynamic
  • Checking that any arguments not statically typed as dynamic match parameter types that are known at compile time
If no candidate passes this test, a compile time error occurs.
Extension methods are not considered candidates for instance method calls, because they are not available during runtime binding.

20.3.3 Conversion to interface types
In C# user defined conversions to interface types are not allowed. For performance purposes conversions of dynamic expressions to interface types are therefore statically rather than dynamically bound. This does have a slight semantic effect, because a dynamic object (i.e. an object whose runtime type implements IDynamicObject) could have given other meaning to the conversion, had it been dynamically bound.

In foreach and using statements (§20.3.4 and §20.3.5), expansions may do dynamic casts to interfaces even though that cannot be done directly from source code.

20.3.4 Dynamic collections in foreach statements
If the collection expression of a foreach statement (§8.8.4) has the static type dynamic, then the collection type is System.IEnumerable, the enumerator type is the interface System.Collections.IEnumerator, and the element type is object.

Furthermore the cast that is part of the foreach expansion is bound dynamically, not statically as is otherwise the case for interface types (§20.3.3). This enables dynamic objects to participate in a foreach loop even if they do not implement the IEnumerable interface directly.

20.3.5 Dynamic resources in using statements
A using statement (§8.13) is allowed to have a dynamic expression as the resource acquisition. More specifically, if the form of resource-acquisition is expression and the type of the expression is dynamic, or if the form of resource-acquisition is local-variable-declaration and the type of the local-variable-declaration is dynamic, then the using statement is allowed.

In this case, the conversion of the expression or local variables to IDisposible occurs before the body of the using statement is executed, to ensure that the conversion does in fact succeed. Furthermore the conversion is bound dynamically, not statically as is otherwise the case for conversion of dynamic to interface types (§20.3.3). This enables dynamic objects to participate in a using statement even if they do not implement the IDisposable interface directly.

A using statement of the form
using (expression) statement
where the type of expression is dynamic, is expanded as:
{ IDisposable __d = (IDisposable)expression // as a dynamic cast
try {
statement;
}
finally {
if (__d != null) __d.Dispose();
}
}
A using statement of the form
using (dynamic resource = expression) statement
is expanded as:
{
dynamic resource = expression;
IDisposable __d = (IDisposable)resource // as a dynamic cast
try {
statement;
}
finally {
if (__d != null) __d.Dispose();
}
}
In either expansion, the resource variable is read-only and the __d variable is invisible in the embedded statement. Also, as already mentioned, the cast to IDisposable is bound dynamically.

20.3.6 Compound operators
Compound operators x binop= y are bound as if expanded to the form x = x binop y, but both the binop and assignment operations, if bound dynamically, are specially marked as coming from a compound assignment.

At runtime if all the following are true:
  • x is of the form d.X where d is of type dynamic
  • the runtime type of d declares X to have type T or T? where T is a primitive type
  • the result of x binop y has the runtime type S where S is a primitive type
  • S and T are either both integral types (sbyte, byte, short, ushort, int, uint, long or ulong) or both floating point types (float or double)
  • S is implicitly convertible to T
Then the result of x binop y is explicitly converted to T before being assigned. This is to mimic the corresponding behavior of statically bound compound assignments, which will explicitly convert the result of a primitive binop to the type of the left hand side variable (§7.16.2).
For dynamically bound += and -= the compiler will emit a call to check dynamically whether the left hand side of the += or -= operator is an event. If so, the dynamic operation will be resolved as an event subscription or unsubscription instead of through the expansion above.

20.4 Runtime semantics of dynamic binding
Unless specified otherwise in the following, dynamic binding is performed at runtime and generally proceeds as follows:
  • If the receiver is a dynamic object – i.e., implements an implementation-specific interface that we shall refer to as IDynamicObject – the object itself programmatically defines the resolution of the operations performed on it.
  • Otherwise the operation gets resolved at runtime in the same way as it would have at compile time, using the runtime type of any constituent value statically typed as dynamic and the compile time type of any other constituent value.
    • If a constituent value derives from a literal, the dynamic binding is able to take that into account. For instance, some conversions are available only on literals.
    • If a constituent value of static type dynamic has the runtime value null, it will be treated as if the literal null was used.
    • Extension method invocations will not be considered – the set of available extension methods at the site of the call is not preserved for the runtime binding to use.
  • If the runtime binding of the operation succeeds, the operation is immediately performed,otherwise a runtime error occurs.
In reality the runtime binder will make heavy use of caching techniques in order to avoid the performance overhead of binding on each call. However, the observed behavior is the same as described here.

The rest of the document covers the other features in C# 4.0: Named and Optional Arguments, COM Interoperability, and Co- and Contravariance. Particularly COM Interoperability also makes use of dynamic:
COM methods are often designed for a language environment that is more dynamic than C#. This means that they will often return weakly typed results, and rely on the calling language to dynamically look up further operations on those results.

IronPython and IronRuby: Four New Releases

There have been four new releases of IronPython and IronRuby; two new ones for each project.

The first pair of releases are for the CTP (Community Technology Preview) of .NET 4.0 Beta 1. Part of the Dynamic Language Runtime is included in .NET 4 and on top of this is built one of the major new features of C# 4.0 and VB.NET 10; the dynamic keyword. dynamic is a new static type and informs the compiler that operations on dynamic objects should be performed at runtime rather than bound at compile time. In particular this makes late bound COM and interacting with objects from DLR languages much more pleasant. It also enables true duck typing (in fact dynamic typing) in these traditionally statically compiled languages.

To go alongside the .NET 4.0 CTP are new releases of IronPython and IronRuby, targetting the same version of the DLR included the CTP.
This is our second preview of IronRuby running under the Dynamic Language Runtime that is built directly into a .NET 4.0 release! As before, this release allows you to use IronRuby objects and types as .NET 4.0 dynamic objects from within C# and Visual Basic code. These binaries are roughly equivalent to the functionality found in IronRuby 0.3. Please also note that the IronRuby CTP for .NET 4.0 Beta 1 will run only under .NET 4.0 Beta 1.
We’re quite pleased to announce the release of IronPython 2.6 CTP for .NET 4.0 Beta 1. This is our second preview of IronPython running under the Dynamic Language Runtime that is built directly into a .NET 4.0 release! As before, this release allows you to use IronPython objects and types as .NET 4.0 dynamic objects from within C# and Visual Basic code. While this release does share a bit in common with the upcoming IronPython 2.6 Beta 1 release (e.g., a number of MSI improvements), the core functionality is essentially that of IronPython 2.6 Alpha 1. Please also note that IronPython 2.6 CTP for .NET 4.0 Beta 1 will run only under .NET 4.0 Beta 1.
This walkthrough shows how the new Dynamic feature in C# 4.0 and Visual Basic 10.0 enables the natural use of object models that expect their host language to provide dynamic dispatch. This capability is demonstrated for an IronPython scenario of accessing libraries that are written for the Python language. Note that more advanced IronPython scenarios that involve hosting your own scripts with access to your application’s object model are beyond the scope of this walkthrough.

Notes:
  • Using Dynamic with Office programming is demonstrated in the Office Programmability walkthrough.
  • Using Dynamic with Silverlight will be shown in a later preview release of Visual Studio 2010.

More importantly (for those using IronPython and IronRuby) we have also had new released with new functionality. The IronPython release is the first beta of IronPython 2.6 and targets compatibility with Python 2.6. As well as new features from Python 2.6 it also includes important new features like ctypes and Python stack frame (sys._getframe) support.
Staying true to our "Conference-driven development schedule," this release is in conjunction with RailsConf '09, where IronRuby was demonstrated running real Rails applications and deploying them on IIS. A ton of work has been done in this release to get real Ruby programs running well, like RubyGems, Rake, Sinatra, and Rails.
We’re pleased to announce the release of IronPython 2.6 Beta 1. As you might imagine, this release is all about supporting new CPython 2.6 features such as the ‘bytes’ and ‘bytearray’ types (PEP 3112), decorators for classes (PEP 3129), advanced string formatting (PEP 3101), etc and as such includes nearly all of the new Python 2.6 features. The small number of remaining features will be implemented for Beta 2. The minimum .NET version required for this release is the same as IronPython 2.0; namely .NET 2.0 Service Pack 1.

There’ve also been several new features added: ctypes, sys.getframe, pyc for pre-compilation, and improved help. Ctypes provides access to C-based APIs and provides a way for Python developers to optimize performance or access libraries that would otherwise be unavailable. sys.getframe is commonly used by Python programs to introspect the call stack and can now be enabled with the –X:Frames and –X:FullFrames command line options; for better compatibility with programs that check for the presence of sys.getframe it is now not available unless these options are provided. The pyc.py pre-compiler is now shipped with IronPython and is available in the Tools\Scripts directory. Finally the help of many builtin functions and types has been improved by including IronPython.xml and IronPython.Modules.xml.

Another focus has been on improving startup time of large applications. This release accomplishes that through the adaptive compilation feature announced with 2.6 Alpha 1 and also includes many other enhancements to reduce the amount of code generation required to start applications. But only do these improvements help large applications but they also significantly speed up basic hello worlds startup time.

There’s also a few minor changes since IronPython 2.0.1 that are worth calling out here:
  • Binaries are ngen’d for both 32-bit and 64-bit platforms on install by default
  • IronPython.msi now offers a little more selection with respect to what you’d like to install. For example, Silverlight templates are optional
  • The default installation location of IronPython.msi no longer indicates whether the 2.6 release is an Alpha, Beta, or a patched release. Future IronPython 2.6 installations will replace previous 2.6 releases which will be uninstalled automatically
  • The -X:PreferComInteropAssembly flag has been removed. All COM interop is now done through normal COM dispatch
There are two big new features in this release. The first is our implementation of the ctypes module. The ctypes module is like P/Invoke for Python. It allows Python code to call into unmanaged DLL functions. Here, for example, I’m calling into the standard wprintf function from msvcrt.dll

IronPython 2.6 Beta 1 (2.6.0.10) on .NET 2.0.50727.4918
>>> import ctypes
>>> libc = ctypes.cdll.msvcrt
>>> ret = libc.wprintf("%s\n", "hello")
hello

Between ctypes and Ironclad, I think we’ll eventually be able to load most native Python extensions in IronPython. Woot!

The other big new feature in this release is a real implementation of sys._getframe. _getframe lets you write code that inspects the Python callstack.
IronPython 2.6 Beta 1 is the second IronPython release today! I'm happy to say that the sys._getframe issues of IronPython 2.6 Alpha 1 have been resolved, and we have re-added urllib.py and other modules dependent upon collections.py to the MSI installer. Man am I ever ready for the long weekend=)

Tuesday, May 19, 2009

Resolver One, Digipede, Timetric & Ironclad

Resolver One is the programmable .NET spreadsheet system created by Resolver Systems. It is of particular interest to this blog because it is written in IronPython and has the largest IronPython codebase in production. You may have heard me mention it here once or twice before.

One of the advantages of Resolver One is how easy it is to integrate with other systems, whether they're .NET libraries or online services. We've recently worked with two companies and produced some demos of this in action.
Digipede is a distributed network computing platform for .NET. The coming version of Resolver One has changes allowing you to distribute spreadsheet tasks across a Digipede grid. If you have version 1.5 (join the beta program here) then you can use this sample to try it out.
The Digipede Network is a brilliant way to get distributed, parallel computation on Windows. It only took a few minor changes to convert Resolver One to run on the Digipede Network and to get the IronPython sample to execute Resolver One workbooks.

Try doing that with a spreadsheet or grid that isn’t based on .NET . . .
Timetric is an exciting new website and web service for storing, searching, graphing and publishing the world’s statistical data. It stores data as time series and lets you view graphs of the data plus manipulate and combine them in lots of interesting ways. As well as making a huge amount of public data available you can also push your own data sets to Timetric and use their services to analyse them.

We’ve created an example of using Resolver One as a client program for working with Timetric data and for searching their public data. You can fetch and view a chart of individual datasets and search their large bank of statistics.

As well as accessing Timetric this example shows:
  • Using the Python Oauth library from IronPython
  • Working with the Timetric API
  • Fetching and parsing CSV files
  • Accessing an opensearch API
  • Parsing XML (the Atom format responses to search queries)
  • Drawing charts with ZedGraph
  • Creating and selecting worksheets from button handlers
  • Using MessageBox dialogs from user code
Resolver One 1.5 will be released very soon. In the meantime here are some other pieces of news on Resolver One and Ironclad:
The Python console is the biggest new feature in Resolver One. It is a great tool, and in this 4minute screencast Kamil Dworakowski demonstrates its features and how to interact with the spreadsheet from the console.
Ironclad is an open source project by Resolver Systems. It is a compatibility layer for IronPython that allows you to use Python C extensions. Thousands of the Numpy and SciPy tests pass on IronPython with Ironclad and it provides the Numpy integration in Resolver One. We ran a competition to find a logo for Ironclad and the one linked to above was the winner.
Greg Bray was the winner of the April round of our spreadsheet competition with his Texas Holdem Monte Carlo Simulator. He has written a review of working with Resolver One:
Resolver One is a new program that aims to make things easier by building the spreadsheet using the IronPython 2.0 scripting language. IronPython is an implementation of the Python scripting language built on top of the Microsoft .NET CLR and DLR runtimes, which means that it is able to easily interact with code from C# and other .NET languages and can make use of any existing .NET library or assembly. It also can perform complex operations such as web service calls, interact with external files or databases, and make use of the Ironclad NumPy library for complex data manipulation. Since it is based on the .NET framework you can embed custom IronPython or .NET objects into the cells instead of just storing numbers, dates and text. This can be very helpful as it allows you to apply Object Orientated Programming techniques to create powerful and easy to maintain spreadsheet.

More IronPython Podcasts and Videos

Another collection of podcasts and videos on IronPython and the DLR, quite a few this time.
These are all Microsoft videos and cover:
  • Implementing a Dynamic Language
  • Inside the Dynamic Language Runtime
  • IronPython for the C# Developer
  • IronPython for the Visual Basic .Net Developer
  • IronRuby for the C# Developer
  • IronRuby for the Visual Basic .Net Developer
  • The Microsoft Dynamic Language Runtime Defined
Harry Pierson (Microsoft IronPython PM) has recorded a new podcast with Deep Fried Bytes. He talked for so long they had to split it into two episodes!
After a few months of hunting him down, Keith and Woody sat down with Harry Pierson who is a Program Manager on the Visual Studio languages team. Harry’s big passion is currently IronPython but he also works with all of the dynamic languages at Microsoft. In addition we discussed the Dynamic Language Runtime and Harry’s views about Open Source and Microsoft.
A few weeks ago I linked to videos of the IronPython related presentations from PyCon 2009 (plus three videos from our IronPython tutorial). If you prefer podcasts to videos then I have some good news for you; the audio has been extracted and turned into a series of podcasts:
Direct links to the IronPython related talks as mp3 files:

Monday, May 18, 2009

IronPython in SharpDevelop 3.1 and Converting C# to IronPython

SharpDevelop is an open source Windows IDE for the .NET framework. MonoDevelop, the Mono IDE, started life as a fork of SharpDevelop.

SharpDevelop lists amongst its features:
  • Support for Mono
  • Supported languages include C#, VB.NET, Boo, IronPython and F#
  • IDE features like refactoring, code completion, Windows Forms designer
  • Can run from a USB device
  • Read only projects
  • Parallel build support for multi-core machines
  • Multi-framework targetting (.NET 2.0, 3.0 and 3.5)
  • Integrated support for tools like FxCop, Wix, NUnit, TortoiseSVN, MSBuild etc
One area where it distinguishes itself from Visual Studio, the 10 000 pound gorilla in the world of Windows IDEs, is its excellent support for IronPython. This just got a lot better in their latest release; 3.1 beta.

Matt Ward, who I believe is responsible for the IronPython support, has written a couple of blog entries on the new features:
Support for designing Windows Forms in IronPython is now available in SharpDevelop 3.1. The original IronPython forms designer was removed when SharpDevelop 3.0 began supporting IronPython 2.0 which had removed support for generating IronPython code from Microsoft's CodeDOM. The forms designer has now been re-implemented to use the IronPython abstract syntax tree (AST) and no longer relies on the CodeDOM.
This is extremely interesting as the lack of CodeDOM support in IronPython 2.0 is part of the reason why IronPython Studio is stuck with IronPython 1. The blog entry demonstrates how to use the IronPython forms designer (with copious screenshots) along with outlining its limitations and how it works. This means that the IronPython support in SharpDevelop far surpasses anything available for Visual Studio or MonoDevelop and gives both Microsoft & Mono a real incentive to improve. I will be trying out SharpDevelop as I am intrigued as to how good their intellisense is for Python; something which is difficult for dynamic languages, but some IDEs do a very good job.
SharpDevelop 3.1 now supports converting C# and VB.NET code to IronPython. It can convert a single file or an entire project. The code to convert between these languages is still under development and has some limitations.

Converting code to IronPython was originally supported in SharpDevelop 2.2 and was based on converting code to a Microsoft CodeDOM and then getting IronPython 1.0 to generate the Python code. In IronPython 2.0 this CodeDOM support was removed so the code conversion feature was removed from SharpDevelop 3.0 since that was using IronPython 2.0. In SharpDevelop 3.1 the code conversion has been rewritten to no longer use the CodeDOM support. It now works by executing the following simple steps:
  1. The C# or VB.NET code is parsed using SharpDevelop's parsing library NRefactory and an abstract syntax tree (AST) is generated.
  2. A visitor class then walks this AST and generates Python code which is added to a StringBuilder.
  3. Once the visit is complete the generated Python code is then displayed or saved to disk.
This is a really interesting feature. Whilst it can't, and doesn't attempt, to cover every possibility it could be very useful for converting example code into an IronPython equivalent. The news was picked up by InfoQ in their Moving to IronPython article and also by Dror Helper (the author of the article covered in my last entry) who recommends using SharpDevelop for developing with IronPython: SharpDevelop 3.1 Beta released – with some IronPython goodness.

Getting Started with IronPython - Part 2: Let's write some code

A follow up from Part 1: where to start, this is a blog entry by Dror Helper on starting programming with IronPython. This entry mainly explores Python the language; including exceptions, classes, methods and magic methods. He has set himself the task of implementing a version of the Mancala board game.
Playing Mancala is easy:

The board has two stores (also called Mancala) one for each player and the goal of the game is to collect as many stones as possible.

remark – if you don’t see the stores in the board above don’t worry my board doesn’t have any. instead I keep the seeds/stones won at the sides of the board.

Each player has six pots he can play from (the ones closer to his side) in the board above my side is the six lower pots.
Players alternate turns, in his turn a player choose one of his pots, takes all of the stones from that pot and places them one by one on every pot he passes in counter clockwise order. When the player passes his store (Mancala) he places a stone there as well.

If the last stone placed in the player’s store he gets another turn.

If the last stone is placed in an empty pot he captures the stones on the opposite pot and places them in his store along with the capturing stone.

And finally the game continues until one of the players clears all of his side (and cannot “move”).

Classes of the game
I have four classes in my Mancala implementation:
  1. Board – has most of the game logic along with the board representation
  2. Store – represent the player’s stores and has a counter for the number of seeds in store
  3. Pot – represent a simple pot along with the amount of seeds in it
  4. MancalaException – game specific exception
In the next episode he promises to look at unit testing in IronPython by showing us how he tested this code.

Sunday, May 17, 2009

A Good Mix 6: Databases, the DLR and Programming Languages

Another selection of blog entries and articles on IronPython culled from the last few weeks.
I last reported on FBConsole in August 2007, when version 0.7.2 was released. FBConsole is a "Firebird RDBMS Tool Powered by IronPython" (a Python console for the Firebird database). This new release (via) is built with IronPython 2.0.
Another blog entry in Japanese. It is useful though as it shows you the ScriptRuntime configuration (app.config) needed for hosting both IronPython and IronRuby in a single application. Useful if you are using the DLR to provide user scripting and want to give your users a choice of languages to use.
Another update, with accompanying release, for the Cobra Programming Language. Cobra is a .NET programming language with both dynamic and static binding, a Python inspired syntax and first class support for unit tests and contracts. As usual there are language, library and documentation updates - including contributions from the community.
Keelkit describes itself as:
KeelKit a domestic ORM tools, automatic generation Model, automatic generation of form, automatically generate unit test framework ,for automatic generation of language exchange with other Model, using the T DBHelper, strongly typed query! No language and database boundaries!
Through the use of an add-on package Keelkit has IronPython support, including integration with IronPython Studio. The latest blog entry (in Chinese) has some of the details. It is available for download from the codeplex site.
Jon Skeet is a well known .NET developer and author. He is also the Stack Overflow user with the highest reputation. In this blog entry he asks whether the new proliferation of programming languages is a land of polyglots or a tower of babel?
What does this mean for the average developer? Currently, if you're writing a non-web application in .NET, you really only need to know a single language – and any of them will do. (Plus potentially SQL of course…) Compare this with web developers who have to be intimately familiar with HTML, CSS and JavaScript – and the differences between various implementations.

How long will it be before backend developers are expected to know a dynamic language, a static OO language and a functional language? Does the benefit of mixing several languages in a project worth the impedance mismatch and the increased skillset requirements? I'm not going to make any predictions on that front – I can certainly see the benefits of each of these approaches in certain situations. They've been designed to play well together, but there are bound to be limitations and oddities: times when you need to change how you write your F# so that it's easily callable from C#, for example.

Whether or not you learn multiple languages to a professional level is one thing, but becoming familiar with them is a different matter. In the course of co-authoring Functional Programming for the Real World (where "co-author" is a bit of a stretch title - I've played more of an editorial role really, with the added bonus of picking on Tomas whenever I felt he was perhaps a little harsh towards C#) I've learned to appreciate many of F#'s qualities, but I don't really know the language. If someone asked me to write a complete application in it (rather than just a toy experiment) I'd be reaching for books every other minute. I hope I'll learn more over the course of time, but I doubt that I'll ever be sufficiently experienced in it to put it on my CV. The same goes for IronPython, although I'm considerably more likely to need Python at work than I am F#. (Python is one of the three "approved" languages at Google, along with Java and C++.) None of this means that time spent in these languages is wasted: I'll be able to apply a lot of what I've learned about F# to my C# coding, even if it will make me pine for things like pattern matching and asynchronous workflows periodically.

The Pyjama Project and PyjamaScheme

The Pyjama project is an open source sub-project run by the Institute for Personal Robots in Education, which is partly funded by Microsoft Research.
The Pyjama Project is a framework for computing. At its core is an integrated editor and interactive console for writing and exploring computer science through dynamic languages. It is designed to be a simple, yet powerful, integrated development environment (IDE) for students, teachers, researchers, and regular humans, too. It runs on most any operating system, including Linux, Mac OS X, and Windows, and is built using .NET and Mono. All sources for the Pyjama Project are open and free---freely available and you are free to use them in various ways. All source code conforms to OSI approved licenses.

Pyjama is built using the Dynamic Language Runtime (DLR) system from Microsoft. This means that languages (such as Python and Ruby) can share data and use a set of common libraries. For example, rather than writing a library for Python, one can write one library (in a portable way) and it will be usable from Python, Ruby, or any of a number of languages without making any changes. Generally, any .NET/Mono library can be turned into a dynamically loaded module for any DLR language. Pyjama was initially designed to control robots interactively, but it can be used for many tasks.

Pyjama is currently under development. It contains many components:
  • Myro - library and infrastructure for controlling robots
  • Editor - simple, but powerful customizable editor
  • Shell - integrated languages
  • Graphics - 2D and 3D
Other Libraries - including neural networks, music, and art
Pyjama is planned to be ready to sample Summer 2009.
The PyjamaDevelopment Page has a screenshot of the IDE so far and instructions for building for Mono or on Windows with Visual Studio Express. It says this of the languages you can use for developing for Pyjama:
The Pyjama interface is written in C#. This will allow any of the .NET/Mono Languages (IronPython, IronRuby, PyjamaScheme, JavaScript, F#, J#, etc) to use the API.
PyjamaScheme is a new language being developed for the project. As far as I can tell it isn't yet a DLR based language, but this is the intention for the future:
Pyjama Scheme is a new implementation of a Scheme-based language for .NET/Mono. It implements many core Scheme functions, but also adds some functionality to bring it into line with the other modern languages like Python and Ruby. The goal is to have a complete environment for teaching, called Pyjama.
Pyjama Scheme is actually written in Scheme. It is written in a special form (continuation-passing style) and then automatically converted through a series of transformations, into C# so that it can be run on .NET/Mono. The generated language could be any language, but for this project we chose Scheme (because we need a Scheme for Pyjama).

Pyjama Scheme is currently under development. Plans:
  1. get a basic scheme working in .NET/Mono [done]
  2. use the Dynamic Language Runtime (DLR) for environments
  3. integrate it with the rest of the DLR
Pyjama Scheme is not a regular scheme and is not meant to be compatible with others. It is designed for education and research.

Friday, May 15, 2009

An IronPython Presentation

Tarn Barford is another .NET developer interested in IronPython. He has just done a presentation for the Victoria .NET user group and the slides and his notes are on his blog:
Python
Basic information about the language and its background. Would have liked to have added meta programming or at least “...” to the paradigms.

I think it’s a fantastic language so I’ll mention it’s expressive, fun, productive or whatever adjectives I’ve got in my head at the time.

People using Python
I really want to demonstrate that Python is real language, being used by real people, to write real code.

I probably should also have mentioned Google App Engine here which I’ve been tinkering with a bit recently and think is pretty cool.

IronPython
The Python language on .Net. Most of the demo is doing cool stuff in .Net with the Python language. I think it goes further than this and allows developers to use their own paradigms and patterns to do .NET development.

C# and IronPython comparison
I saw this example in the awesome IronPython in Action book. I included it as it’s a great introduction IronPython code, as it clearly and simply shows differences on similarities with C#.

Getting a Jump Start with IronPython

Darrell Hawley is a Microsoft C# MVP and .NET developer with an interest in IronPython. He has written an article for Packt Publishing on getting started with IronPython:
As a .NET developer, you’ve probably heard IronPython mentioned in a blog post or an article, but do you know what it is? IronPython is Microsoft’s implementation of the Python language. Python is known for readability and its’ proponents claim that applications written with Python are done faster, use fewer lines of code and are more maintainable than those written using more traditional languages. The great thing about IronPython is that you get the advantages of the language, while being able to leverage your knowledge of the .NET framework. In this article by Darrell Hawley, we will cover a few basic aspects of IronPython and use them to create a Windows Form.
The article covers using IronPython from the command line and with a code editor like Scite, working with .NET regular expressions, Python classes and Windows Forms.

Thursday, May 14, 2009

New Sample: Writing a DLR Language in C# or IronPython

As well as being the basis for language implementations like IronRuby and IronPython the Dynamic Language Runtime is intended to be a framework making it easy for YOU to implement a dynamic language that runs on .NET and Mono.

So far there hasn't been a rush of languages using the DLR. The only ones I'm aware of are:
  • IronScheme (at 1.0 Beta 3a released May 2nd 2009)
  • Groovy DLR (at 1.0 Beta 4 released May 9th but may already be abandoned - see my previous blog entry)
  • Nua (Lua for the DLR) (no releases and last checkin July 2007 - so looks dead)
  • DLR Basic (no releases and last checkin November 2008)
  • IronSmalltalk (at version 0.1 with last checkin October 2008)
  • IronLogo (at version 0.0.1 with last checkin January 2008)
  • Managed JScript (Javascript) - developed by Microsoft and only ever released as Silverlight binaries. The future of Managed JScript is very unclear.
Python, Ruby, Javascript (Managed JScript), Scheme, Groovy, Lua, Smalltalk, Basic and Logo would be an impressive set of languages for the DLR; but of these only Python, Ruby and Scheme look as if they are under active development - and only IronScheme is a non-Microsoft project. However, various people have blogged about writing your own programming language with the DLR or using the DLR as an expression evaluator for custom DSLs so there is a lot of experimentation going on which may yet bear fruit.

Importantly there is a new kid on the block. The DLR has always shipped with a very small example language called Toyscript. This has only ever demonstrated a subset of what is involved in using the DLR. Bill Chiles (DLR program manager for Microsoft) has just announced on the IronPython mailing list a new example language called Sympl.

Not only does it include walkthrough documentation, but the language implementation is done in both C# and IronPython. Yep, it is possible to write DLR based languages in Python which is not something I expected.

Dino (core IronPython developer) says of Sympl:
One of the really cool things about Sympl that I think is worth emphasizing is the version that’s implemented in IronPython. Not only is it the 1st language I know of implemented in IronPython but it’s able to do cool stuff like create IDynamicMetaObjectProvider implementation from Python objects which are already dynamic objects! Now we just need to get someone interested in writing IpyIpy.
If you're interested in Sympl it is included in the DLR sources:
Here is the full announcement from Bill:
There’s a new DLR sample to which I wanted to draw folks’ attentions. There is a small language implementation sample, implemented in both IronPython and C#, with an accompanying walkthrough document. The code is now on the DLR Codeplex site under ...\Languages\Sympl, and the document is on the “Documents and Specs” page which you can get to from the list of links on the home page. Under the Sympl directory is also an examples directory with some code written in Sympl, which all runs when you execute Main in program.cs. The code and document will be in zip files we will produce to go along with the VS Beta1 release coming soon.

Sympl demonstrates how to implement a very simple language using the Dynamic Language Runtime (DLR) as it ships in .NET 4.0. The goal is to make you aware of how to get started building a language on the DLR. Sympl does not show production quality rich .NET interoperability and binding logic. It does walk you through the following:
  • Using DLR Expression Trees (which include LINQ Expression Trees v1) for code generation
  • Using DLR dynamic dispatch caching
  • Building runtime binders
  • Building dynamic objects and dynamic meta-objects
  • Supporting dynamic language/object interoperability
  • Very simple hosting with application supplied global objects
  • Basic arithmetic and comparison operators with fast DLR dispatch caching
  • Control flow
  • Name binding within Sympl
  • Method and top-level function invocation with fast DLR dispatch caching
  • Very simple .NET interoperability for runtime method and operation binding
  • Using the DLR’s built-in COM binding
  • Closures
  • Assignment with various left-hand-side expressions
Sympl does not yet demonstrate using any Codeplex-only DLR code such as the DefaultBinder, the faster and nicer .NET reflection trackers (Sympl’s is painfully slow and simple), or the common DLR hosting (yet).

A few caveats on the documentation. First, the walk through document still needs some polishing, but I wanted to get it out to people. Feel free to send me comments or ask questions. Second, the document refers to a couple of other documents on our Codeplex site, but we haven’t updated some of them since DEC 08. Those will be updated in the next couple of weeks. I think the sympl.doc walkthrough is very readable on its own and has all the relevant code snippets included in it as well. Lastly, while the example is overall reasonably simple, it does demonstrate all the basic concepts of building a language on the DLR with explanations, and therefore, is quite long. However, the document unfolds in the same way the Sympl language implementation evolved, so it progresses well for incrementally increasing your awareness of how to build a language on the DLR.

Hope you enjoy and find this code and doc helpful!

Groovy: A New Language on the DLR?

There was an interesting new release on a codeplex project a few days ago:
Groovy is an "agile dynamic language for the Java Virtual Machine", with Java inspired syntax and capabilities but with additional dynamic features "inspired by languages like Python, Ruby and Smalltalk". Groovy has a Rails inspired web framework called Grails and aims to be the "JVM's defacto dynamic language", but faces stiff competition from both JRuby and Jython.

So if Groovy is a JVM language, what is it doing on .NET and the Dynamic Language Runtime?

Groovy DLR is the brainchild of Gavin Grover. His website says:
Groovy/DLR
Recently, Gavin Grover began adding the Groovy Language to Microsoft's dynamic language runtime. He's focusing on creating a full-featured syntactic skin using combinator parsers because that's what's most lacking in (J)Groovy. Groovy's syntax is the next generation in the C language family, aiming to bring self-referentiality to the language syntax.
He talks about the project in his latest blog entry:
I've just put the source code for beta-04 of Groovy-DLR 1.0 online, with some improvements, including evaluating code in separate files, and in-place comment lex-definitions. Time for a brief roadmap...
The roadmap then continues with a discussion of features that Gavin intends to add to Groovy. They don't seem to be specific to Groovy DLR, and in fact at the end of the blog entry he says:
One last thing...
Oh yes, I almost forgot to mention, I'm not doing any of this in C#. I'm switching languages, back to the JVM, back to the (J)Groovy AST, to SCALA.

...

The resulting code will be called Groovier. It'll be Apache 2.0 licensed to encourage the (J)Groovy developers to copy it for bundling with Groovy. Groovier will be to Scala and the GrAST what the (J)Groovy Language is to Java and the JVM. Groovy is getting groovier and Groovier.

Although it isn't entirely clear it looks like Groovy DLR is still-born (a dead-end, deceased) and will see no future development unless someone else picks it up. Shame.

One comment buried in the middle of the entry is interesting:
Python uses indentation to group statements into blocks, C-syntax languages use curlies. Groovy will provide both techniques simultaneously, using symbol {:. When working, we'll abstract the technique to other lists in the syntax using symbols (: and [:. Groovy developers will have the choice of syntactic forms, even able to mix both styles together in the same portion of code, just like in natural language.
Indentation for block structure in Python is a stroke of genius and has already influenced languages like Haskell and F#. Nice to see it spreading further.