Tuesday, October 28, 2008

IronPython 2 and the Fifth Assembly

IronPython 2 uses Dynamic Language Runtime expression trees, which have been merged with LINQ expression trees. These have extension methods and IronPython itself also uses extension methods. These need the .NET 3.5 compiler, but if you provide an 'ExtensionAttribute' type then the compiled assemblies still work with .NET 2 (extension methods are effectively a compiler trick).

Unfortunately the 'ExtensionAttribute' type supplied with IronPython 2 clashes with the version in .NET 3.5 if your project also references the System.Core assembly.

The latest release candidate of IronPython solves this problem by providing a fifth assembly. Harry Pierson (DevHawk) gives us the details: The important details are:

In IronPython 2.0 Beta 5, there were four DLLs that implement IronPython:
  • IronPython.dll
  • IronPython.Modules.dll
  • Microsoft.Scripting.dll
  • Microsoft.Scripting.Core.dll
In our RC1 release, we’ve added “The Fifth Assembly”: Microsoft.Scripting.ExtensionAttribute.dll. As you might guess from its name, it has only a single public type: ExtensionAttribute. By having ExtensionAttribute in its own dedicated assembly, we can avoid the type collision at compile time by not referencing both System.Core.dll and Microsoft.Scripting.ExtensionAttribute.dll in the same project.

In IronPython, we reference the ExtensionAttribute assembly because we use the C# 3.0 complier but IPy has to be able to run on .NET Framework 2.0 SP1. However, projects that embed IronPython in a .NET 3.5 project (aka C# 3.0 or VB 9.0) will reference System.Core instead. The only reason why you would explicitly use the ExtensionAttribute assembly was that if you, like us, wanted to build your app with .NET 3.5, use extension methods but still be compatible with .NET 2.0 SP1. We’re guessing there aren’t many of our customers doing that, but if you are, explicitly referencing ExtensionAttribute will work just as it does for compiling IronPython itself.

It’s important to remember two things about the Fifth Assembly:
  1. Never reference System.Core and Microsoft.Scripting.ExtensionAttribute in the same project.
  2. Always deploy Microsoft.Scripting.ExtensionAttribute.dll as part of any solution that embeds IronPython (or IronRuby or vanilla DLR for that matter), even if you don’t reference it explicitly within your project.
Unfortunately this still doesn't solve the problem for ASP.NET projects that use .NET 3.5. The valiant IronPython team found another solution, and Curt posted the workaround to the IronPython mailing list:

Here's what we recommend as a workaround.

The CodePlex work item that tracks this request now contains a modified version of Microsoft.Scripting.ExtensionAttribute.dll. For this one particular case, where simultaneous referencing of the original version of our assembly and System.Core.dll causes problems, you can replace the "shipping version" of our DLL with the modified version. This will eliminate the "is ambiguous" error message.

The magic that makes this work is called "type forwarding". The updated DLL simply contains a type forwarder to System.Core for the type System.Runtime.CompilerServices.ExtensionAttribute. As a result, there is no second copy of that class, and the CLR loader redirects all references from the rest of IronPython to System.Core.

This file, though strongly-named and signed, is even more "not supported" than the rest of IronPython :). By this I mean primarily that we haven't run any test passes using the new assembly, and don't plan to. However, as it doesn't actually contain any code, the risks from this modified version should be very small.

We hope to address this problem differently for the next release, but our promise to stay compatible with .NET 2.0 for this release has somewhat constrained the type of solution we are able to offer.

Thanks for your patience!

Oslo's Intellipad Scripted with IronPython

The Microsoft PDC conference is in full swing and there are various pieces of IronPython / Dynamic Language Runtime related news adding to the backlog of things I need to post. (C# 4 becoming a dynamic language through the DLR, preview of IronPython integration in Visual Studio 10 and so on.)

One interesting snippet is that the Intellipad tool, part of the Oslo framework, is scripted with IronPython.

Oslo is a modelling platform "that aim to significantly simplify designing, building, managing and scaling service-oriented and composite applications that can span from the enterprise to the Internet". It includes a new declarative language called "M" "for working with data and building domain models".

UPDATE: Martin Fowler has a new blog entry covering Oslo in more depth.

Intellipad is inspired by Emacs, and in fact had the codename "Emacs.NET". It is user scripted with IronPython instead of Lisp. Online material is hard to find, but this page has some screenshots.

It looks like Intellipad is useful outside of its Oslo home. Dody Gunawinata described it as "a WPF editor with IronPython crack".
From the readme:

Customizing commands

Almost all commands that are available in Intellipad have been written in Python using the object model exposed by the application. The Python files are scattered inside the Settings directory.

Commands.py contains most of the commands for Intellipad.

Configuration specific commands are placed in their respective directories (Emacs or VI or VisualStudio).

A command definition consists of three parts.
  • A "Executed" function definition, that acts as the command handler and provides the logic for the command
  • An optional "CanExecute" function definition, that determines when the command is enabled
  • A command wireup, that is done by calling the "Common.Command" function. This is the where the Executed and CanExecute are wired together along with the Command Name and default key binding

For the moment, a restart is required for the changes to take effect.

Friday, October 24, 2008

IronPython 2 RC1 Released

IronPython 2.0 Release Candidate 1 has just been released. IronPython 2 is the version of IronPython that is built on the Dynamic Language Runtime, and targets Python 2.5.

Along with the new release candidate the team notes: If no major issues with the RC are reported, we hope to ship the final 2.0 release in around a month’s time. Anyone planning to move to 2.0 please try the RC and let us know of any issues you find.
As well as fixing around 40 bugs (making more than 500 bug reports that have been closed since IronPython 2 went into alpha), there are a few new things in this release.

Firstly, the MSI installer now has an option to ngen (pre-JIT) the installed binaries. Ngen’ing the dlls gives a significant startup performance boost. The option is turned off by default but if no issues are reported, it will be enabled by default in a future release.

The last couple of betas of IronPython 2 have included the Python 2.5 standard library in the msi installer - making the standard library available to your IronPython code without having to do a separate install or set 'IRONPYTHONPATH'. In this release standard library modules that can't be used from IronPython are not included (ones that rely on C extension modules). There is a list of the excluded modules and packages, along with the reasons for their exclusion, at: Python Standard Library Not Included in the msi Installer

This release also attempts to solve the name collisions with 'ExtensionAttribute' in .NET 3.5. More about that in a separate entry.

Check out the release notes for a full list of the bugfixes in this release. One of the fixes that we are particularly pleased with at Resolver Systems is that with 'clr.CompileModules' it is now possible to compile pure Python packages into binary assemblies (or executables). We have an experimental binary distribution of Resolver One that runs on IronPython 2 and our source code tree consisting of hundreds of Python modules compiles down to a handful of assemblies. Like Python 'pyc' files these import faster as the parsing and compiling is already done. As a nice bonus the generated assemblies can also be ngen'd for even faster importing.

The experimental 'Resolver One on IronPython 2' can also use numpy through our IronClad project, allowing you to do interesting matrix calculations with huge amounts of data inside a spreadsheet. As the performance profile of IronPython 2 is very different, and Resolver One is heavily optimised for IronPython 1, it will be sometime before you see it in a release - but be sure I'll post news here as it starts to happen.

This release still includes the example DLR language 'ToyScript', but now in a separate zipfile of its own. When IronPython 2.0 final is released, a new home for ToyScript will need to be found.

Wednesday, October 22, 2008

Pumping Iron Talk

This Saturday Harry Pierson (Program Manager for IronPython) is giving a talk on Dynamic Languages on .NET at the Southern California Code Camp Community:
(Calling it SoCal makes me think of William Gibson, but there you go.)

10:00 AM - Saturday, October 25, 2008

IronRuby | IronPython | DLR

As you may know, Microsoft is developing IronPython and IronRuby, .NET implementations of the popular open-source programming languages Python and Ruby. While it’s clear that Microsoft wants to attract existing Python and Ruby developers to .NET, the role of IronPython and IronRuby for existing .NET developers is less clear. What value is there for a .NET developer in learning IronPython? What are the tradeoffs between IronRuby and a more traditional .NET language like C# or VB? Harry Pierson, new PM for IronPython, will discuss where dynamic . languages fit in the.NET developers toolbox.

Tuesday, October 21, 2008

Crack .NET: Runtime Debugging & Scripting Tool

Crack.NET is a runtime debugging and scripting tool that gives you access to the internals of any .NET desktop application running on your computer. Crack.NET allows you to walk the managed heap of another .NET application, inspect all kinds of values on objects, and even manipulate those objects via IronPython scripts.
Once you find an object or type that you're interested in, you can manipulate it, and others, by writing and executing an IronPython script.

Crack.NET was built against version 1.1.2 of IronPython.DLL and IronMath.DLL.

IronScheme turns One

1.0 Beta 1 that is. Yesterday IronScheme 1.0 Beta 1 was released.

IronScheme is a rewrite of IronLisp, and aims to be an R6RS conforming implementation of Scheme based on the Microsoft Dynamic Language Runtime.

This new language implementation allows you to write .NET applications in Scheme or embed Scheme as a scripting language. It also opens up the possibility of Python and Scheme (and other dynamic languages) interoperating through the DLR.

This release includes a Visual Studio (SP1) plugin and is the creation of the author of the Open Source .NET xacc ide which has Scheme support.
From the 1.0 Beta 1 release notes:

Notably lacking for R6RS
  • call/cc only supports outward continuations
  • all raise's are continuable in the sense they can be caught
  • no tail calls in dynamic-wind
  • a generic load procedure for toplevel R6RS files
  • many extras procedures and macros in (ironscheme) library, mostly provided by psyntax
  • syntactic CLR integration, automatic conversion of closures into delegates
  • Abdulaziz Ghuloum and Kent Dybvig for psyntax (and friends)
  • Derick Eddington for making SRFI-1 R6RS compatible
  • Philip L. Bewig for SRFI-41
  • Marc Feeley for pretty-print
  • anyone else I missed
Planned to-do's (before final release)
  • testing of anything else
  • any additional CLR integration required
  • documentation
Updated (fixed/new)
  • Visual Studio 2008 SP1 plugin
  • A tiny MVC web framework
  • Added a lot of documentation, more to come
  • Many bug fixes
  • See 'Source Code' page for more details

Monday, October 20, 2008

IronPython Updates

I've got a backlog of links to post; normal service will be resumed shortly... In the meantime here a few snippets that may be of interest.

Through the 'Pyc' compiler sample, which uses 'clr.CompileModules' under the hood, IronPython 2 can compile Python code to binary assemblies. These assemblies can be ngen'd (pre-JITed) making imports roughly three times as fast as from a Python sourcecode file. This is great because IronPython import performance is sloooow...

In the latest beta (2 beta 5) compiling packages was broken, meaning that we couldn't build and test a binary distribution of Resolver One running on IronPython 2. This problem was supposed to have been fixed in the latest codeplex source code drop, so today I tested it out.

The first thing I had to do was build IronPython 2 from sources on a machine that has .NET 3.5 installed, but not Visual Studio 2008. It took me a while to find precisely the right magic command line invocation to build in release mode rather than debug, so here it is:

>C:\Windows\Microsoft.NET\Framework\v3.5\MSBuild /p:configuration=release IronPython.sln

It is the property ('/p:') setting that sets the build configuration to release instead of the default debug build.

The good news is that compiling packages works. Even better, compiling packages with sub-packages works. Unfortunately, compiling packages with sub-packages with sub-sub-packages is broken - so we still can't test a binary distribution of Resolver One. At least there is time for it to be fixed before the final release.

Speaking of final release, there is still no official date but there was a hint of a rumour of something being prepared for PDC (October 27-30th) so it may not be far off.

Curt Hagenlocher (IronCurt) let some more good news slip on the mailing list today. Apparently the RC1 installer for IronPython "will have the option of GACing and/or NGENing IronPython". Cool.

Monday, October 06, 2008

Compiling the DLR for Silverlight and IronPython with Silverlight RC0

Silverlight Release Candidate 0 is out, this means that the old versions of IronPython and the Dynamic Language Runtime don't work with the new version.

Silverlight includes a cut down version of the .NET framework called the Core-CLR. IronPython and DLR have to be compiled against the Core-CLR assemblies for the version of Silverlight you are targeting.

But IronPython is Open Source, so you can just compile it against the new Silverlight assemblies right? Jimmy Schementi explains:
"Of course! Everything should just work, since there were no major breaking changes in Silverlight that affect the DLR between Beta2 and RC0. So, you hacked up the csproj files to point at mscorlib.dll, system.dll, etc in the new Silverlight install directory (C:\Program Files\Microsoft Silverlight\2.0.30923.0), compile, and it builds fine. Then you try to run an app ..."

Next you need to make a simple change to your dynamic application to tell it about the new version of the runtime. Jimmy covers this in his blog entry as well. Of course rather than compiling IronPython yourself, it is easier to wait for Jimmy to release a new version of the SDK.
Jimmy also explains the changes in the DLR hosting API if you are hosting IronPython or IronRuby in your Silverlight application.

Friday, October 03, 2008

OpenBabel from IronPython

Noel O'Blog reports on the recent OpenBabel release that explicitly supports IronPython.
OpenBabel is:

"a chemical toolbox designed to speak the many languages of chemical data. It's an open, collaborative project allowing anyone to search, convert, analyze, or store data from molecular modeling, chemistry, solid-state materials, biochemistry, or related areas."

The OBDotNet release comes with instructions on how to use it from IronPython:

First of all, unzip the OBDotNet distribution, and set the environment variable BABEL_DATADIR to point to the data folder contained within.

If running IronPython in the unzipped OBDotNet folder, you can use the following to initialise OBDotNet:

>>> import clr
>>> clr.AddReference("OBDotNet")

Otherwise, you need to need to:
(1) add the OBDotNet distribution folder to the PATH environment variable
(2) include the full path to the OBDotNet.dll when adding the reference, as follows (note the 'r' at the start):

>>> import clr
>>> path = r"C:\Tools\OpenBabel\ob-22x\windows-vc2005\OBCSharp\OBDotNet-0.1\OBDotNet.dll"
>>> clr.AddReferenceToFileAndPath(path)

Once the reference to OBDotNet has been added, you can import and use the OpenBabel bindings just like from CPython:

>>> import OpenBabel as ob
>>> conv = ob.OBConversion()
>>> conv.SetInFormat("smi")
>>> mol = ob.OBMol()
>>> conv.ReadString(mol, "CCC")
>>> mol.GetMolWt()