Thursday, March 26, 2009

IronPython 2.6alpha 1 Released

Dave Fugate has just announced the release of IronPython 2.6 alpha 1.
There is also a page comparing performance of the IronPython release with Python 2.6.1. The performance shows a degradation because the new adaptive compilation techniques are very much a work in progress:
The announcement:

We’re pleased to announce the release of IronPython 2.6 Alpha 1. As you might imagine, this release is all about supporting new CPython 2.6 features such as the ‘bytes’ and ‘bytearray’ types (PEP 3112), decorators for classes (PEP 3129), advanced string formatting (PEP 3101), etc. The minimum .NET version required for this release is the same as IronPython 2.0; namely .NET 2.0 Service Pack 1. Unlike the 2.0 series of IronPython, we plan to release only a couple Alphas and Betas of IronPython 2.6. As such, it’s key that we get your feedback on the release(s) quickly to incorporate requested changes.

Besides CPython 2.6 features, another significant change in this release is that ipy.exe now uses “adaptive compilation” by default. Adaptive compilation is a technique in which IronPython:

1. Interprets and executes Python method calls up to N times for a given method. If you’re only going to execute a method a few times, it’s typically faster to interpret the method instead of compiling and executing it

2. Compiles and executes the Python method call on the N+1 invocation of the method. Compilation of a Python method is a heavyweight operation, but we can reuse the result for subsequent invocations

3. Reuses the previously compiled method for new calls to the Python method. This operation is much faster than interpreting the method call as the method was already compiled in the previous step

The reason for this change is that it provides a nice performance gain for Python code containing lots of functions/methods that only get called a few times. All this said, this feature is still undergoing active development and as a consequence some Python scripts may actually run slower with it turned on. For this reason, our old default mode of running Python scripts is still available by passing the –O or -D flags to ipy.exe. Any feedback on how this new feature affects your IronPython applications performance-wise would be greatly appreciated.

There’s also a few minor changes since IronPython 2.0.1 that are worth calling out here:
  • IronPython.msi now installs NGEN’ed binaries by default
  • IronPython.msi now offers a little more selection with respect to what you’d like to install. For example, Silverlight templates are optional
  • The default installation location of IronPython.msi no longer indicates whether the 2.6 release is an Alpha, Beta, or a patched release. Future IronPython 2.6 installations will replace previous 2.6 releases which will be uninstalled automatically
  • The -X:PreferComInteropAssembly flag has been removed. All COM interop is now done through normal COM dispatch
You can download IronPython 2.6 Alpha 1.

The IronPython Team

Tuesday, March 24, 2009

The Coding Geekette's Book Reviews: IronPython in Action

Sarah Dutkiewicz is the Coding Geekette, a .NET developer who also like Python and has presented at many .NET community events on IronPython. Sarah is speaking at PyCon 2009 on FePy, the community distribution of IronPython: Pumping Iron into Python: Intro to FePy.

In a recent blog entry she reviews IronPython in Action:
Overall, I would recommend IronPython in Action for anyone wanting to learn IronPython. The examples in this book were easy to follow and very applicable to everyday programming. Even if you're an experienced IronPython programmer, IronPython in Action would be great to have on hand as a reference. I'm looking forward to buying the final copy once it comes out, just to have as a reference (and to plug in my future IronPython talks).

Exploring ActionScript Bytecode with IronPython

OK, so I'm not fully certain what this about - but it certainly looks interesting.

AbcExplorationLib is "a library to read and write ActionScript Byte Code (ABC) files. When completed it could be used for compiled program analysis or as part the back end of an experimental compiler. It is written in F# but it could be used from other .NET languages. Inspiration for this library comes from many excellent libraries for bytecode manipulation such as BCEL, ASM, Cecil or System.Reflection.Emit".

The blog entry linked to here, is about experimenting with ActionScript bytecode from IronPython; and in particular with the bytecode for the switch statement.
Given the following ActionScript code:

var x;
for(x = 1;x <>
switch(x) {
case 1:
print("one");
break;
case 2:
print("two");
break;
case 3:
print("three");
break;
case 4:
print("four");
break;
}
}

We compile this file to a .abc file using the following command:

$ java -jar /opt/flex3sdk/lib/asc.jar testswitch.as

testswitch.abc, 280 bytes written

By using a little IronPython example included with AbcExplorationLib we can see how this library interprets the LookupSwitch opcode:

$ mono /opt/IronPython-2.0/ipy.exe ../ipyexample/abccontents.py testswitch.abc
...
Instructions:
getlocal_0
pushscope
pushbyte 1
getglobalscope
swap
setslot 1
jump dest177

And so on...

Monday, March 23, 2009

Numerical Computing in IronPython (with and without Ironclad)

John Cook posted a couple of entries on getting started with IronPython, which I linked to in the last IronPython-URLs selection post. Since he made those entries he's been posting a bit more on IronPython, and specifically numerical computing with Python and IronPython.
John has written an article on CodeProject showing standalone code for computing normal probabilities in IronPython.
The algorithm is based on formula 7.1.26 from Handbook of Mathematical Functions by Abramowitz and Stegun, affectionately known as "A&S." The algorithm given in A&S is for computing the error function erf(x). The error function is related to the the function Phi by a simple transformation. (The corresponding function in SciPy is scipy.special.erf.)

Because this code has no dependencies other than the standard math module, it should run anywhere Python runs. It has been tested on Python 2.5, Python 3.0, and IronPython. The code should be accurate to seven decimal places.

Now that we have an implementation of phi(x), we can write a simple wrapper around it to compute normal probabilities.
The code he uses in the article is available for download.

The main reason John needed to write this stand alone code is because the normal libraries for this kind of numerical computing in Python are SciPy and Numpy. These have various parts written in speed, and therefore don't work with IronPython which is written in C# rather than C.

However, there is one way that these extensions can be made to work with IronPython. This is through Ironclad, a compatibility layer that mimics the Python 2.5 C API and allows Python extensions written in C to be used from IronPython 2. Ironclad is an open source project created by Resolver Systems, to allow the use of numpy with Resolver One - a spreadsheet system written in IronPython and programmable through Python.

As of recent releases of Ironclad it successfully runs thousands of the SciPy and Numpy tests, and many other C extensions can also be used. John invited William Reade, the lead developer of Ironclad, to write a guest post explaining how Ironclad works and what it does:
The upshot of my recent work is that you can now download Ironclad, type ‘import ironclad; import scipy‘ in an IronPython console, and it will Just Work. I am programmer, hear me roar!

Hundreds of tests now pass in both NumPy and SciPy, and I hope that some of you will be inspired to test it against your own requirements. For example, the Gaussian error function has been mentioned a few times on this blog (and, crucially, I have a vague idea of what it actually is), and I can demonstrate that scipy.special.erf works perfectly under Ironclad.
In the article William explains how to setup Ironclad, gives examples of what works and how to use it, and also discusses the limitations and future directions for the Ironclad project.

An interesting project occurs to me. It would be fun, but I haven't the bandwidth to attempt it at the moment. It should be easy to write a wrapper around Ironclad so that Python extensions like SciPy and Numpy could be used from IronRuby or even from C# and F#...

Extending a .NET Application with the DLR: 3 Articles

Three different blog entries on extending an application by embedding IronPython and the Dynamic Language Runtime.
Creating a simple calculator program and an evaluator app that lets you enter code into a textbox and press a button to execute. Both embed IronPython and are available for download. These simple examples demonstrate how easy it is to add an embedded scripting language in a C# application.
Another simple example, but this one more directly aimed at those wanting to add scripting support into their applications:
There’s another interesting application for IronPython and IronRuby: adding scripting support for your existing .NET applications. This can be a very useful and powerful way to extend your applications and give the user freedom to program their own mini programs, scripts or whatever in your applications. It could be good for defining rules, assigning and calculating values, etc.

Usage of this class is pretty simple. You have to provide the object the script you want to execute and the input variables the script will have available as local variables. Once this is done, you have to call the Execute method, and this method will either return the output variables of the execution of the resulting script, or throw an exception.
This article is on... embedding IronPython and IronRuby. It's very short, but it does show the use of the ScriptRuntime (to get access to the clr module for the hosted engine).
For a more complete introduction to embedding IronPython and the DLR, you can either get IronPython in Action or read my embedding articles.

Sunday, March 22, 2009

Binding dynamic types in WPF using IronPython and DLR

There have been a couple of projects in the past integrating IronPython into WPF XAML to add more dynamic features:
WPF, Windows Presentation Foundation, is the new User Interface library by Microsoft which is part of .NET 3.0. It uses the graphics card GPU to do UI drawing, so as well as being more powerful than Windows Forms it can also perform better. It isn't available in Mono though, so has the disadvantage of not being cross-platform. XAML is the XML markup used to describe GUIs, transformations, etc in WPF and Silverlight (which uses a subset of WPF XAML).

Here's (yet) another way of adding dynamic features to WPF. This time it is for dynamic binding:
Covered in this post:
  • Creating classes in IronPython and comparing with C# classes. Adding public properties in IronPython class.
  • Using classes/interfaces in IronPython class which is in the hosting assembly. For example, if the application hosting the DLR and executing the IronPython class at runtime has an interface IDyn, this article covers on how to use that interface within the Python script.
  • Accessing the dynamic type created in the IronPython and making an instance of it.
  • Finally, binding the dynamic type to WPF controls.

Another Awesome Collection - IronPython Links

A selection of IronPython and Dynamic Language Runtime related web pages and blog entries.
John Cook makes a start with IronPython. This means using one of the three distributions available from the Codeplex homepage: the source code, the prebuilt binaries or the msi installer.
Having got started John Cook is disappointed that he can't use Python C extensions from IronPython, and finds it hard to call into IronPython from C#.

The answer to the latter problem is to use the IronPython hosting API, and John Cook has written another blog post on the former problem which will get an IronPython-URLs entry all of its own.
This seems like an exciting announcement, but is short on links to find further information. The upshot is that NaroCAD has, or will shortly have, support for scripting it with IronPython 2 and possible IronRuby as well:
This means that with minimal change right now NaroCad will support in the same way any DLR capable language (like IronRuby).

What can you do right now with the IronPython integration?
  • You may execute IronPython commands like this "Hello world" program:
  • clr.AddReference("System.Windows.Forms")
  • from System.Windows.Forms import *
  • MessageBox.Show("hello from NaroCad")
  • You may: execute the lines you write, load a file from disk, or save onto a disk your hard worked program
  • Pressing "Enter" to any line will execute your line of program, and in case the program will misbehave the exception will be shown to the user
  • You can change all the properties of the command line editor like this: panel.CommandLineEditor.ShowLineNumbers = 1; or anything that reflects the CommandLineView public members (so you can hide/show the items in real time). The way to react with NaroCad internal API is a subject of change but this is a fully working concept
  • The editor use the file: commandLineHistory.py (or create a blank one if none exists), as the editor default text.
Darrell Hawley, who has recently been experimenting with IronPython and nose, had difficulties when experimenting with the IronPython hosting API. If you add a reference to one set of assemblies in your C# project (and use the classes they contain), but add a reference to a different copy of the same assembly in the IronPython part of your project, then you are going to get odd errors (and odder error messages)!

You can avoid these problems altogether by adding the reference to the hosted IronPython runtime programattically from C# (instead of calling clr.AddReference from IronPython):

Assembly assembly = typeof(CSharpClass.Person).Assembly;
runtime.LoadAssembly(assembly);
A pair of Argentinian blog entries by the same author, which I think bewails various problems that the writer had with IronPython 2 - specifically startup time and compiling to binary with the Pyc IronPython compiler.

Unfortunately he is correct. IronPython startup time, especially when importing a lot of Python modules, is not great. There are workarounds, and one of them is compiling your application and dependent modules to binary using Pyc (the writer reports a 30% improvement for importing the Cheetah package).

The writer had been led to believe that this process was transparent to your application which isn't quite the case:
  • To import from compiled assemblies you need to add references to them first
  • You have to manually add references to System.dll and mscorlib.dll which you don't have to do with IronPython scripts. This should be fixed in a future release of Pyc.
  • Another bug in Pyc means that if you compile scripts to executables then sys.argv may not be correctly populated with command line arguments (oops!).
At Resolver Systems we compile and deploy a large application using Pyc. We don't use Pyc for the main executable, but instead have a custom executable that embeds IronPython and launches the application.

Whilst there are bugs, the authors rant in the second entry that IronPython compilation isn't identical to CPython bytecode compilation is a little misplaced. Import performance on the other hand is something that the IronPython developers do need to focus on. Thankfully they say that they are addressing this, and that we will reap the benefits of this in IronPython 2.6.
Chris Cavanagh wants to implement an expression language, similar to the formula language in Excel, using Dynamic Expression Trees. Expression Trees are used to represent Asbract Syntax Trees (AST) in dynamic languages built on the DLR, but it looks like Chris has managed to parse expressions and generate charts without actually using the DLR. The promised part 2 hasn't yet followed, but hopefully soon...

UPDATE: In a comment Chris points out that part 1 does use the DLR to evaluate expressions, and that part 2 is now up!
A short entry from Harry Pierson with a syntax highlighting definition for Python for CodeHTMLer. CodeHTMLer is a simple program that translates plain text code into a colorized HTML version of the code, useful for posting code examples to a blog. Harry also has a language definition for F#, a statically typed functional programming language for .NET.

Friday, March 20, 2009

Herding Code Episode 37

Herding Code is "a weekly podcast with K. Scott Allen, Kevin Dente, Scott Koon, and Jon Galloway". They have a .NET focus, and episode 37 is an interview with Jon Udell - who is a technical evangelist for Microsoft and a Python / IronPython enthusiast.
This week on Herding Code, Jon leads a talk with Microsoft Technical Evangelist Jon Udell, about strategies for Internet citizens. That is, making public information available for retrieval and manipulation through structured data feeds and Internet standards. The group discusses related topics like digital identity and OpenID and shares their thoughts on Oslo, DSLs, dynamic languages like IronPython.

Jeff Hardy Reviews IronPython in Action

Jeff Hardy is the creator of NWSGI. This is a .NET implementation of the Web Server Gateway Interface protocol; a Python protocol for web applications. His goal is to be able to use Python applications and frameworks like Trac and Django from IronPython with the Microsoft IIS server. He also blogs about his progress in getting Django and setuptools to run under IronPython.

As with other recent entries on this news blog, he is featured here because he has posted a review of IronPython in Action:

He says:

I wasn't really sure what to expect when given this book. Having read it, I now see that it is exactly what it says it is – IronPython, in action. For any serious work with IronPython (more than just using it as an alternative Python interpreter), than this book will save you a ton of learning time. The examples are clear, concise, and – most importantly – realistic. Even if you already know Python and .Net inside and out, but are new to IronPython, this book will save you some headaches learning about how they interact.

This was exactly our goal, so we're pleased!

Thursday, March 19, 2009

Using Nose with IronPython

nose is a testing library for Python. nose is a popular alternative to unittest, which is part of the standard library, and boasts features like:
  • Test autodiscovery
  • Output capture
  • Not restricted to class based tests
  • Class and module level setUp and tearDown
  • Assert introspection
  • Test generators
  • Plugin mechanism
Personally I don't find the object oriented way of organising tests a problem; test methods grouped in classes seems very logical to me. I also like assert methods and the richer error messages they provide. I usually end up defining all sorts of convenience assert methods myself that do the heavy lifting of my testing anyway. I'm sceptical of the magic that libraries like nose and py.test do to provide some of the same information with bare asserts - and that magic doesn't work with IronPython anyway.

The test discovery stuff is invaluable though. Whenever I create a new project I inevitably end up recreating some variation that does test discovery for the project with unittest. It looks like there is a strong move to get test discovery for unittest into the standard library. Unfortunately it is the sort of subject that tends to attract bike shed discussions so it may take a while! Unless someone is actively working on it I will implement something myself (for inclusion in the standard library or not).

Because of the magic that nose does, plus some abortive early experiments when IronPython was much younger, I've always assumed that nose doesn't work with IronPython. Thankfully Darrell Hawley has mostly proved me wrong. There is a big performance problem, but it does work:
I just spent the past week at the 2009 Microsoft MVP Summit where I got a chance to talk to the some of the crew behind the dynamic languages at Microsoft. When I mentioned that I had gotten nose, a Python unit testing framework, working with IronPython, I was strongly encouraged to blog about it. Without further adieu, here's my code.

import sys
sys.path.append(r"C:\Python25\Lib\site-packages\nose-0.10.4-py2.5.egg")
sys.path.append(r"C:\python25\lib")
sys.path.append(r"C:\Python25\Lib\email")
sys.path.append(r"C:\Python25\Scripts")
import nose
nose.main()

I dropped this script into a directory with a couple of bogus unit tests and ran it with Python 2.5.4. Results were consistent with my expectations - successful tests passed and failing tests threw exceptions. When I ran the same suite against IronPython 2.0.0, I still received the correct results but the script took 1 minute and 20 seconds to complete. Compare that to less than a second (estimated) for CPython.

The root of the problem appears to be the large number of GeneratorExitExceptions being thrown. I'm not sure why they're being thrown, but I'll continue to investigate and report my findings.

Writing an IronPython Debugger

One of the things that programmers in static languages complain about working with dynamic languages is the lack of tools. Many of them are unaware of the tools that are available for languages like Python:
And these are just the ones I have used or am aware of. I often think that what proponents of static languages mean by "the tools aren't available" is really "I want to use Visual Studio". This is just me being cynical though.

For IronPython the story is genuinely not so good. IronPython doesn't have Python stack frames; a performance optimisation that reduces the function calling overhead which is very high in the C implementation of Python (CPython). Unfortunately this means that core APIs in the sys module, particularly settrace and setprofile, are unavailable in IronPython.

You can use both IronPython Studio and Visual Studio to debug IronPython code, but the 'traditional Python tools' for debugging and profiling don't work. Because IronPython classes, objects and scopes have a different representation to .NET objects created in C# or VB.NET the debugging experience even in Visual Studio is not as good as it could be (although still useful).

The good news is that Dino Veihland (core developer) says that Python stack frames, along with sys.settrace and at least some of its friends, will be coming to IronPython soon. As there is a performance cost (~10% in general Dino estimates) it will be optional, but you will be able to enable it when you need debugging support or for compatibility reasons.

In the meantime, Harry Pierson (Microsoft PM for IronPython) has been experimenting with creating a custom debugger for IronPython built on top of the .NET debugging tools and APIs. He is basing his work on the .NET command line debugger MDbg for which the sourcecode is available. He has also had some assistance from Mike Stall, something of a guru in .NET debugging.

One of Harry's goals with ipydbg, is to make it Just My Code aware so that he can usefully set breakpoints step through Python code and skip all the uninteresting DLR stack frames that exist when IronPython is executing. He has made good progress (which you can download and try for yourself of course), and these blog entries chart what he has done and still has left to do:
Adding JMC support makes ipydbg significantly more usable. It’s almost like a real tool now, isn’t it?

Wednesday, March 18, 2009

An IronPython on Silverlight Collection

A collection of links about using IronPython with Silverlight, the Microsoft 'flash killer' browser plugin.
A post by the MetaDeveloper to get you started developing Silverlight applications with IronPython and the Silverlight Dynamic Languages SDK. It includes the development process using Chiron.
Whilst we're talking about the Dynamic Languages SDK for Silverlight, it's time for a word from the guy responsible for maintaining it: Jimmy Schementi. This is a long post that describes the SDK and answers some Frequently Asked Questions on the subject.

The post starts with:
It would seem the next release will be focused around REPLs, testing, scripting support, and maybe some server-side integration pieces. And you’d be right. Proof is in the project’s public source code repository.
And ends with:
I hope that made some things clear; the DLR, IronRuby, and IronPython are committed to supporting Silverlight developers. Unfortunately, the reverse is less strong, so if this is important to you please talk about it on the silverlight.net forums. Things might be a bit nicer when the inner ring of the DLR (Microsoft.Scripting.Core.dll) is part of the System.Core in Silverlight, but we’ll see.
Definitely worth reading the bits in the middle!
Another MetaDeveloper post, this time describing a Silverlight library that I'd not heard of: SilverlightFX. Apparently, "Silverlight.FX is an application framework for building Rich Internet Applications with Silverlight 2. It focuses on providing building blocks and controls that allows applications to be naturally well-architected".

In this post David shows how to create an IronPython application that uses the SilverlightFX libraries, and shows off some of its declarative XAML goodness.
Thanks to comments from Jimmy Schementi and Richard Klafter I've updated a couple of my IronPython and Silverlight articles. Specifically, the articles on communicating between Javascript and IronPython with the Scriptable attributes and embedding IronPython in a C# application have both been improved.
This is the announcement from Miguel de Icaza about the release of Moonlight 1.0. Moonlight is the Mono implementation of Silverlight for Linux which is being developed with support from Microsoft. Moonlight 1.0 is the equivalent of Silverlight 1.0 - a rendering engine with a Javascript API. It doesn't include the CoreCLR which is the really interesting bit of Silverlight and what allows it to be programmed with IronPython. This is in Silverlight 2.0, and coming in Moonlight 2.0, but in the blog entry Miguel describes the history and progress of Moonlight project - including a screenshot from a very fancy looking 3D visualization written in Python.

You can track the progress of Moonlight 2.0 on its Demo Status Page - which includes some IronPython and IronRuby demos and how well they do or don't work on Moonlight.
I haven't tried this, I don't know what's in it, or even if IronPython works with it - but a beta of Silverlight 3 is now available! The article linked to is on Ars Technica and it looks like that various Silverlight 3 beta 1 downloads (including runtime, SDK and tools) are available for download in advance of the keynote by Scott Guthrie and Bill Buxton at the MIX09 conference currently on in Vegas.

From what I've heard Silverlight 3 isn't a revolutionary change, but adds requested features and fixes annoyances with Silverlight 2. This sounds like the right move as Silverlight is still gaining adoption (although making good progress) and developers are still getting to grips with it.

Personally I find the internet a much more pleasant place with Flash turned off - but there are some useful web applications that use it. The programming model for Silverlight is great, I've really enjoyed using it and being able to program in the browser with Python is a major plus. I just hope we don't start seeing Silverlight used for serving adverts...

Searching for Tweets with IronPython

Posts by Saveen Reddy have featured regularly on this blog, most recently his series of posts on automating Visio from Powershell and IronPython. He is back, and this time he is having fun with the Twitter Search API, using IronPython to search for tweets.

This simple example shows you how simple it is to search for tweets using Twitter’s Search API.

Key techniques demonstrated in this sample code

  • Using System.Net.WebClient to download a file
  • Searching for nodes in System.Xml using namespaces
  • Simple ATOM-feed parsing

Tuesday, March 17, 2009

Writing to Freebase with IronPython

Freebase is "A social database about things you know and love". It has a public API to access and update structured data. The API is based on the Metaweb Query Language (MQL) and naturally there are Python bindings to the API.

The API Guy has a blog entry about writing to Freebase using IronPython, including code samples:
My mom was in town visiting this weekend, and when I went to demo Freebase for her, I asked her to name a famous person. She suggested Pope John Paul II, so we looked him up. Most of the information that you would expect Freebase to know about him, it did. But there was one glaring omission: it didn’t know that he was Catholic.

Since Freebase knows 263 people who have ever been a professional Pope, I decided that it would take a little too long to update them each by hand. And so I set out to scribble up a few lines of Python to use the Freebase API libs for Python.
In order to use the Freebase Python lib from IronPython you need the simplejson library.

Resolver One 1.4: One Day Special Offer (66% off) and Unit Testing Spreadsheets

Resolver One 1.4 was released a while back, and we have some great new examples. These include integration with the R statistical language, embedding the Resolver One calculation engine into IronPython programs, unit testing spreadsheets and using Numpy inside the spreadsheet.

I was going to leave posting on all of this for a few days (still working my way through the backlog - I figure by now that the backlog is permanent though), but we have a one day special offer for Resolver One. 66% off ($75) on March 17th 2009 only.

If you've been considering splashing out on Resolver One, now is the time to grab it:
For those who have absolutely no idea what I'm talking about, but somehow still carried on reading this far, first of all thanks. Secondly, Resolver One is the .NET and Python powered programmable spreadsheet created by Resolver Systems. Resolver One has the largest known codebase of IronPython anywhere (around 40 000 lines of code in production plus around 130 000 or so in the test framework). Resolver One brings a better (and more Pythonic) programming model to creating spreadsheets and spreadsheet systems.

Resolver One 1.4 was recently released. The major new features are that Resolver One is now built on IronPython 2, Numpy integration, model side scripting and a bunch more spreadsheet functions. You can read all about it, along with the latest winner of the Resolver Spreadsheet Challenge which integrates the R Statistical Programming Language with the Resolver One spreadsheet UI. You can read all about it in my blog post:
We've been posting new examples up on the Resolver Exchange showing you how to get the most from Resolver One. Here are some of the latest:
A simple demo of the Resolver One Numpy integration that approximates Pi. (We missed Pi day for this one - shame.) You'll need to read our docs on Numpy integration to use it. If you're running 64 bit Windows then this may be of interest too.

Using Numpy inside the spreadsheet grid (even with arrays holding a million numbers) totally rocks. See this short screencast for a demo. Mad props to William Reade and his work on Ironclad that makes it possible.
Using the new Formula property on cells to import CSV files into Resolver One.
Part of what makes Resolver One unique is that every spreadsheet is an IronPython program - with your own code in as well (the Dreamweaver of spreadsheets is one way it has been described). The new RunWorkbook function allows you to integrate spreadsheets together, but can also be used directly from IronPython (without the UI) allowing you to use the Resolver One calculation engine from Python or .NET programs. This demo / library shows you how.
One possible use case for the IronPython integration is to unit test your spreadsheets - or even use TDD (Test Driven Development) to create them! This is an example of testing a spreadsheet from the Python standard library test framework unittest. This would actually be fantastic to bring to the mainstream spreadsheet world and I intend to write an article to go with the demo soon.
An example of integration between Resolver One and the Frontline Systems .NET optimisation libray, using their platform SDK. Also a good example if you are interested in using this library from IronPython. Combining Solver libraries with traditional spreadsheets with Excel is pretty common, so nice to see that you can also do it with Resolver One.
Even without the special offer, you can get a free 31 day trial of Resolver One, and if you are sharing the spreadsheets you create or using it within an Open Source project then you can request a free license.

Monday, March 16, 2009

Winforms with Mono and IronPython Tutorial Series

IronPython runs great on Mono, and if you use the official Mono installer then IronPython is included out of the box. (Although the version of IronPython 2 is quite old now and could really do with being updated - when I tried it the latest Windows binary worked fine on Mac OS X which isn't something you hear often. For Debian and Ubuntu users it is as easy as apt-get ironpython.)

So cross platform application development with IronPython is really possible, and for client applications Windows Forms is the GUI toolkit to use. Although it looks ungainly by default on non-Windows operating systems, with customization it is possible to create applications that look great on Windows, Linux and the Mac.

Up on ZetCode there is a 12 part tutorial on creating applications with IronPython and Windows Forms on Mono. It covers the basics, plus dialogs, advanced use of controls, drag and drop and handling painting yourself. It even creates Snake and Tetris applications as examples! Anyway - looks really good.
This is Mono IronPython Winforms tutorial. Mono IronPython Winforms tutorial is for beginner programmers. The goal of this tutorial is to teach readers basics of GUI programming in Mono Winforms for the IronPython programming language. The tutorial is created and tested on Linux. Nevertheless, it can be used on other operating systems as well. Most examples should run without modification. Images used in this tutorial can be downloaded here.
This tutorial also links to another tutorial on using GTK# with Mono. Although the tutorial is for C# it is usually easy to convert from C# to IronPython once you are used to it (strip type declarations, curly braces and semi-colons and often this is enough!). For Linux apps GTK# is possibly a better choice for the UI and this is one of the goodies that comes with Mono and not with the standard .NET framework.
These two user interface toolkits aren't the only choices with Mono and IronPython. For Mac users there is also Cocoa# (another Mono goodie - here's a blog entry from someone who used it recently). For cross-platform programs you might want to look at Qyoto - the Mono bindings for Qt. Miguel de Icaza (Mono lead developer at Novell) has just linked to an IM (Instant Messenger client) called Synapse built with Mono and Qyoto: Mono and Qt. Looks nice.

Sunday, March 15, 2009

Another Awesome Collection - IronPython Links

Another tab closing session - a lot of links to IronPython (etc) content here. There are a couple of interesting reports of Python libraries working with IronPython, but we start with an IronRuby article:
An excellent article by Ben Hall on testing .NET code (C#) with IronRuby and RSpec.
In this article, I will introduce you to Ruby and IronRuby and demonstrate some basics of Ruby interoperating with Microsoft .NET Framework-based code. I also explain how frameworks such as RSpec can be used to generate examples of how objects are intended to behave, providing both documentation and verification that the system is built correctly. This will set the stage for a future article in which I will explain acceptance testing in detail and demonstrate creating acceptance tests with IronRuby.
A Japanese article on IronPython. No idea what it says!
A demo of an Azure (Microsoft cloud platform) application that hosts the Dynamic Language Runtime and can dynamically execute code in either IronPython or IronRuby - with server side security sandboxing.
The Expert Texture blog gives an early report (but no code) on running IronPython with the Digipede clustering network (for distributed grid computing).
Next step: distribute IronPython objects. Fairly straightforward, but I’ll write a reusable C# Executive to load the IronPython class definition.
A translation of the Microsoft: "IronPython: Engaging the Python Community in its Own Language (pdf)" document. Resolver Systems gets a mention here.
Or "How M.C. Escher Would Have Packaged His Software". A rant on the packaging of IronPython Studio - the IronPython integration with Visual Studio 2008 which is based on the Visual Studio Shell and can also be run as a standalone application. Personally I don't think that IPy Studio is worth the pain and you're much better off with a proper Python IDE like Wing.
Jacob Santos on starting to learn both C# and IronPython:
Well, I didn’t know Python at all, and I was able to create a solution in days. I think that is more a feature of the language, than my mad skillz. If you have been programming before, then you can just “get” Python, because it is just that simple. That is part of the reason I want to further my understanding of it in Visual Studio.

By doing both C# and Python in Visual Studio I have both of the best worlds. Some solutions are easier in Python than in C#, so it makes sense to use Python. However, I still want to build the application with both. That is to say, I don’t want Python scripts sitting there available for people to look at or have to load Python runtime to execute Python scripts. I just want them both to work together.
Another article on IronPython in non-English. This one in some FlashPaper document reader thing that actually makes it hard to read. The article is quite long and looks worth it though.
With the latest version of IronPython it takes only a minor modification to use the PyParsing library. PyParsing is a popular Python parsing library with a novel object oriented approach to constructing grammars.
Director is a Python plugin framework "that allows developers to create command line plugins for tools making it easy to add new functionality". Steve Milner notes:
Even though it's not in the documentation IronPython 2.0.1 passed Director's tests making it the first release that works with IronPython!
An interesting looking project that combines the interactive shell environments of various scripting languages including both Python and IronPython (for both Windows and Windows Mobile - although DLR based languages don't run on Windows Mobile):
MAS is a script-based command shell (CLI) for Microsoft Windows NT (Windows XP, Windows Server 2003, Windows Vista, including 64-bit editions) and Windows CE (Embedded and Windows Mobile for Pocket PC). MAS creates an environment for executing scripts interactively. When you type commands at the MAS prompt, MAS plays the part of the host application, it interact with the selected script engine to get your command executed, and makes objects and services available for the script.

It makes it easy to interact with Automation objects and ActiveX objects and controls.

MAS help file contains a complete user's guide that shows how to use Windows Script Runtime and Windows Script Host components, script the Windows shell, Microsoft Agent assistants, common applications such as Microsoft Office, access databases and files using ADO and MSXML, call Web services using MSSOAP, inspect and modify the computer configuration using WMI (WBEM), encode audio and video using Windows Media Encoder, and many others, in several scripting languages including Microsoft JScript, Microsoft Visual Basic Scripting Edition (VBScript), RubyScript (ActiveRubyScript), PerlScript, Python and ForthScript. Limited functionnality is also available using ObjectRexxScript and PHPScript.

It also includes Majerus.net Active Shell .NET, also known as MASX, which is the .NET Framework + DLR (Dynamic Language Runtime) conterpart of the native COM/OLE Automation + Active Scripting version of MAS. It provides the same experience with .NET DLR script engines and .NET objects, giving full access to the .NET Framework using managed script engines such as Managed JScript and IronPython, and later, Dynamic Visual Basic (VBX) and IronRuby.

IronPython at PyCon and the Python Language Summit

PyCon 2009 - the international Python conference in Chicago - is rapidly approaching.

Harry Pierson (Microsoft IronPython Program Manager) has blogged about how IronPython features this year:
Harry also mentions the difficulties he's had in getting the Microsoft Gold Sponsorship for PyCon actually into the hands of the PSF. (Python Software Foundation - who as well as running the conference also own the copyright on the Python code and the trademarks etc).

As well as talks on tutorials covering IronPython, PyCon starts this year with a Python Language Summit. This is the first time this has been tried:

The Python Language Summit is an invitation-only event for developers of Python implementations and the standard library, to be held at PyCon 2009 in Chicago, on Thursday March 26 2009. The coordinator for this summit is Andrew Kuchling.
Attendees will not deliver formal presentations. Instead, the Summit will be a series of sessions focusing on specific issues. After a 5-10 minute presentation by a session leader, the attendees will concentrate on discussing issues and deciding upon actions to take.
Both Dino and I will be there, with a particular interest in how Python can better support alternative implementations. One of the sessions particularly addresses this issue:
What do the various VMs want/need from CPython to help with their implementations?
  • Marking CPython-specific tests in the test suite?
  • Getting an implementation agnostic test suite for the Python language?
  • Separating the language tests and the pure Python part of the standard library into a separate project? (Or publish them as a separate package.)
  • Transition plans for 3.0?
Session leader: Frank Wierzbicki of Jython.
The language summit isn't the only new addition to PyCon. There is (running concurrently - very topical) a VM summit for those developing Virtual Machines for dynamic languages. Ivan Kristic has details on his blog:
It is a free, full-day, invite-only event intended to bring together people doing cutting edge work on implementing virtual machines for dynamic programming languages. It’s not designed to be a dog-and-pony show but a hardcore, technical working session.

We’ll have representatives of the CPython, PyPy, GemStone MagLev, Rubinius, Mozilla TraceMonkey, Parrot, Sun Da Vinci Machine, Microsoft DLR, Jython and JRuby communities, with several more being approached.
Very exciting, hopefully see you at PyCon.

Thursday, March 05, 2009

Get in the Queue, your locks don't work here

Threading, locking and concurrency are all hot topics at the moment. One place where threading and locking don't work as a way of controlling access to shared resources are in distributed web applications. This blog entry by Tarn Barford discusses the use of Queues from IronPython (with plenty of example code) when working with web apps on the Azure platform.
I am, of course, talking about a pattern in high scale web application design, Queues. More specifically the Queue service on the Windows Azure cloud computing platform.
So when do you use Queues?

Its common to use thread synchronization objects in a web application to serialize access to a resource, but this thread synchronization approach doesn't work so well when you have multiple applications, over multiple servers competing for exclusive access to a global resource.

The first scenario I've found which suits the Queue service is in a user driven content ranking system. I suspect I'll find many more places to use the Queue services as I continue to learn more about developing and designing applications on the cloud platform.

In the web application I'm building I want users to be able to rate dynamic content and I also want to record the number of views. This is a very common concept which allows the users to provide feedback which can then be shared with other users and used to dynamically rank the content.

A Multilanguage REPL (and hosting IronPython in IronRuby)

REPLs (Read Eval Print Loops - interactive programming environments) are all the rage in Redmond these days, and Tomáš Matoušek (core IronRuby developer) isn't immune.

This blog entry uses the DLR hosting API to create a multilanguage REPL using IronRuby. This demonstrates hosting multiple DLR languages in a single application (using the app config file) and hosting IronPython in IronRuby.
DLR Hosting API is a common API for all DLR languages: IronRuby, IronPython, and others. It abstracts away language specific details and provides API that could be used in a language independent manner. The API is primarily designed to ease usage of dynamic languages from strongly typed .NET languages and to make it easier to write applications with a scriptable object model. Nonetheless we can take advantage of the API in a scripting language as well. For that purpose the most interesting features are management of isolated scripting environments (script runtimes) and cross language code execution. I presented the former in my last post. Let’s check out the latter today.

To run examples you’ll need to build the latest IronRuby, IronPython and DLR from sources. You can do so using the latest sources from IronRuby GIT repository or DLR CodePlex site.

Wednesday, March 04, 2009

IronPython in SharpDevelop

SharpDevelop is an open-source .NET IDE for Windows. (The cross-platform Mono .NET IDE MonoDevelop started life as an early fork of SharpDevelop.)

Features of SharpDevelop include:
  • Support for Mono
  • Supported languages include C#, VB.NET, Boo, IronPython and F#
  • IDE features like refactoring, code completion, Windows Forms designer
  • Can run from a USB device
  • Read only projects
  • Parallel build support for multi-core machines
  • Multi-framework targetting (.NET 2.0, 3.0 and 3.5)
  • Integrated support for tools like FxCop, Wix, NUnit, TortoiseSVN, MSBuild etc
Of course on this blog it is the IronPython support that we are interested in.

Back in 2007 Matt Ward blogged about the IronPython AddIn Internals, and some of the features it included. Now that SharpDevelop 3 is out, the support has matured and Matt has done a new blog entry showing how to use the Python standard library with SharpDevelop 3 and IronPython 2:

Make your .NET application extendible using DLR : IronPython

Krishna Vangapandu has posted a blog entry on using the Dynamic Language Runtime hosting API to make a .NET (C#) application extendable:
There are different ways to define this – extensibility in applications, adding macro capability to your applications, providing add-in like facility in .NET applications. For me, all of this mean the same – ability to modify your application (data or the UI) at runtime by means of scripts. Like I mentioned in the previous post, I wanted to dig into DLR or Dynamic Language Runtime which is basically a framework that could be used to provide dynamic languages support on the .NET platform. Even though I am not much interested in giving up C# for dynamic languages like Python or Ruby, I was very much keen to add facility to make my applications extensible using these languages (to tell you the truth, I am more comfortable with JavaScript or Groovy than any of these).

Anyway, I worked on an application which displays a datagrid and one could modify that the data being displayed using Python script at runtime. The way I did it was to host IronPython inside C# using the DLR framework.

The article would be a detailed step by step procedure on each and every piece of code that has been written so that it helps the readers get familiar with the following
  • Using WPF Datagrid and databinding. Also shows Dependency Properties.
  • How to refresh WPF Datagrid items.
  • Hosting IronPython inside C# applications.
  • Passing data to and fro between C# and IronPython script.
If you're interesting in embedding IronPython in .NET applications, for making your application extensible or allowing user scripting, then you may find these articles useful: Embedding IronPython Articles. Of course the best reference on this subject is IronPython in Action.

Monday, March 02, 2009

Rob Oakes on IronPython in Action and Learning to Fish

The manuscript for IronPython in Action is on the way to the printers. In the meantime Rob Oakes has been reading IronPython in Action (the Early Access version) and has posted a review online:
He has a very interesting reaction to the book. He does seem generally positive, here is his final conclusion:
In case it isn’t clear, I heartily recommend that anyone interested in learning Python or .Net purchase this book. It certainly has its idiosyncrasies, but it still manages two roles very successfully: 1) introductory text and 2) reference manual. While novice readers (like myself six months ago) will find tremendous value in reading the book from cover to cover, more experienced hands will find value in skimming individual chapters. As someone who has written technical documentation, I can testify that such a feat is extremely difficult to pull off.
What is unusual is that he took away something from the book that was quite unanticipated by Christian and myself: "While I initially set out with a desire to learn a skill, I instead received an education. In the process, I got excited about programming, Python, and Open Source". Rob feels that instead of just showing how to use IronPython, the book caused him to examine his approach to programming. In fact he starts his review by asking: Is IronPython valuable for people who are new to Python or .Net programming? More to the point, can it be used to teach a novice “how to fish?”.

This reaction wasn't as a result of a deliberate agenda on our part, we simply wanted a book that would be relevant to both Python programmers interested in .NET and .NET programmers interested in dynamic languages. Naturally this meant that the book needed to start with a Python tutorial (chapter 2) and an introduction to .NET (chapter 3) that also shows how IronPython integrates with the underlying platform.

It is however no secret that I'm particularly enamoured of the lightweight approach to object-oriented programming that Python affords. Although the main goals of chapters 4-6 are to introduce new Python features to .NET programmers (lambdas, properties, first-class functions, etc) and new .NET libraries to Python programmers (XML, Windows Forms, .NET serialization, etc), there is a reason I chose a structured application using several object-oriented design patterns as the example.

Developers who have mainly worked with statically typed languages often have an odd idea about dynamic languages; that they make it harder to create large scale maintainable applications. Meanwhile, developers who mainly use dynamically typed languages like Python think the opposite - and feel that creating large projects in more verbose languages is a maintenance nightmare waiting to happen! (Even though the IDE will help you write the code, you typically spend more time reading code than writing code - so verbosity imposes a cognitive burden even if you can write the stuff just as fast.) By showing off the features of both Python and the .NET framework through building a well structured (if rather small) example, I hoped to demonstrate that dynamic languages make clean and elegant designs possible.

Rob ends the review with:
To answer my own question: Can IronPython in Action be used to teach a novice how to “fish?” Most definitely.


Azure Table Storage from IronPython

Azure (meaning the color of a cloudless sky) is Microsoft's new cloud computing platform. Table Storage is the data storage API of Azure. It "offers structured storage in the form of tables. The Table Storage API is a REST API for working with tables and the data that they contain".

Here are two recent blog entries showing how to use Table Storage with / from IronPython:
A blog entry by Jon Udell:
In an earlier installment of the elmcity+azure series, I created an event logger for my Azure service based on SQL Data Services (SDS). The general strategy for that exercise was as follows:
  1. Make a thin wrapper around the REST interface to the query service
  2. Use the available query syntax to produce raw results
  3. Capture the results in generic data structures
  4. Refine the raw results using a dynamic language
Now I’ve repeated that exercise for Azure’s native table storage engine, which is more akin to Amazon’s SimpleDB and Google’s BigTable than to SDS. Over on GitHub I’ve posted the C# interface library, the corresponding tests, and the IronPython wrapper which I’m using in the interactive transcript shown below.
This entry is by Tarn Barford. He writes some C# helpers and demonstrates using them from IronPython:
My goal was to write a bit of scaffolding to make using the storage service fun and easy from in IronPython. The Windows Azure SDK comes with a library in the samples which does the low level work interfacing with the API and provides some nice classes to work with. There are methods to generate table schemas by reflecting on model classes and another sample implements all the standard .NET providers.

I was hoping to write all the scaffolding and the model classes in IronPython but, in the first of a series of set backs, I found the development storage server behaves differently than the cloud. For some reason you need to create tables on the development storage server using a command line tool, passing your model assemblies as arguments. Apparently this will be fixed soon, but trying to stay focused I decided I'd have to write my models in C# for now.

Once I had an assembly with some models I could use the DevTableGen.exe command line tool that comes with the Azure SDK to create tables on my development storage server.

Sunday, March 01, 2009

Embedding IronPython 2 and the Dynamic Language Runtime

My series of articles on embedding IronPython have been updated for the latest version of the hosting API in IronPython 2. As well as being updated there are two new articles, an introduction and overview of extending a C# application with IronPython plugins and an example 'Scripted Turtle' application that allows you to draw funk images through Python scripting:
One of the most impressive things about DLR based languages like IronPython and IronRuby is how easy they are to host in applications to add dynamic features. You can even host IronPython in IronPython, something that Resolver One makes good use of to provide a separate execution context for each spreadsheet.

During the development of IronPython 2 the DLR was changing regularly and each new release seemed to bring breaking changes to the API. This means that most of the online examples, including my own articles, quickly became out of date. I've finally updated all my articles (except for one on creating a custom executable which is clearly marked as being for IronPython 1). There are quite a few different articles and examples for download here, covering various different aspects of embedding IronPython in C#. As the hosting API is common between DLR based languages, most of these articles also apply to IronRuby as well.

ISV Innovation: IronPython for the C# Developer

ISV Innovation are a Microsoft partner who do online training for a wide range of subjects. This includes a series of on-demand web seminars on IronPython and Dynamic Language Runtime related subjects:
The course trainer is Bill Lodin (Vice President of IT Mentors). The description is short but straightforward: "This session provides an introduction to the syntax of IronPython for the Visual C# developer".

This web seminar also links to related topics in the the Microsoft Dynamic Language Runtime for Developers Web Seminar Series:
  • Part: 1 IronPython for the Visual Basic Developer
  • Part: 3 Inside the DLR
  • Part: 4 Implementing a Dynamic Language
  • Part: 5 IronRuby for the Visual Basic Developer
  • Part: 6 IronRuby for the C# Developer