Monday, August 31, 2009

Developing cross-platform applications with IronPython: GtkBuilder and Glade on IronPython

IronPython is a great cross-platform development language, running on the Microsoft .NET framework for Windows and on Mono just about everywhere.

Even for CPython developers IronPython has a few features that make it of interest. These include:
  • Under IronPython there is no Global Interpreter Lock (GIL) meaning that multi-threaded pure-Python applications can scale across multiple CPUs with no extra work from the developer
  • .NET AppDomains allow you to create Python engines and restrict their security privileges, including controlling network and filesystem access and which assemblies the Python code can use
  • Easily create Python applications with multiple isolated Python engines in the same process
  • Easy to compile applications to binary and make binary only applications
  • Through the Dynamic Language Runtime (DLR) you get interoperability with languages like C# / F# / IronRuby and IronScheme
  • Extending IronPython with C#, for performance, is worlds easier than extending Python with C
For desktop application development a big question is what GUI framework to use if you want your application to run cross platform. Mono has a very complete implementation of Windows Forms, which is the standard .NET user interface on Windows. For Windows applications Windows Forms is good looking and easy to use, but by default on Linux and the Mac it isn't very attractive. It is useful for porting Windows applications but wouldn't be your first choice for a native Linux or Mac OS X application.

It definitely is possible to create good looking cross-platform applications with the Mono version of Windows Forms. Have a look at these screenshots of the Plastic SCM for an example. I haven't learned the requisite magic tricks to do this though.

There are at least two other alternatives: Qt (Qyoto) and Gtk (Gtk#).

Unfortunately there aren't yet good tutorials for developing with these user interface toolkits and IronPython, but these two blog entries will get you started with Glade and the GtkBuilder.

Someone asked me if I could add the missing parts of GtkBuilder in Gtk#Beans so he could use it with IronPython on mono.

Hey, it looks there's no missing parts ! It all works fine since day one. Here's the the trick...
Thanks to Stephane for his answer to my query about using GtkBuilder in IronPython. It turns out his Gtk#Beans package provides the magic sauce that is currently missing from the current stable release.

For completeness, here’s the code I sent him that accomplishes the same thing using the older Glade.XML object for those that are interested. It answers a long standing mailing list question about using Glade.XML.Autoconnect in IronPython...

Jonathan Hartley and Brett Cannon Review IronPython in Action

It's good to have friends in high places, and two of them have reviewed IronPython in Action. Both Jonathan Hartley and Brett Cannon are experienced Python developers.

Jonathan Hartley is a colleague of mine at Resolver Systems where we have been working full time with IronPython for the last few years. I finally managed to blackmail him into actually reading IronPython in Action. Despite this he seemed to genuinely enjoy it and has posted a glowing review. The first paragraphs of the review have some interesting things to say about the place of IronPython in the Python world, which is the section I've quoted below:
Having spent some years working with .NET, and with a series of intriguing personal experiments in Python under my belt, I originally approached IronPython some years ago with a modicum of of trepidation. I feared that the weld between the two would be intrusively visible, forming distracting differences from regular Python. I feared for the execution environment, the data types, and perhaps even the syntax itself.

Experience with IronPython showed these worries were needless. I have found IronPython to be a remarkably pleasant marriage – the same elegant language we know and love, given first-class status in the .NET runtime. Gifted with seamless interoperability with other .NET languages, the dowry from such an alliance turns out to be all the .NET libraries in the world, including the substantial and highly capable .NET standard libraries themselves.

IronPython is, to some extent, a niche implementation of a niche language. However, its position seems to potentially be one of importance and strength. Not only does it allow Python programmers to use .NET libraries – and does so admirably, but it also allows the existing legions of .NET programmers to be introduced to the joys of Python. They will fall in love with it, and will be able to introduce it into their workplaces in a way that is politically acceptable. After all, it is now simply another .NET language. Since .NET is orders of magnitude more popular than Python, this could yet turn out to be the most important source of future Python adoption.
Brett Cannon is one of the core Python developers, and despite not developing on Windows he has a keen interest in alternative implementations of Python and has reviewed IronPython in Action:
To the point: if you need to program for Windows and you want to use Python, you should get IronPython in Action. The book does a good job of walking you through examples covering all the major APIs and tools a Windows programmer will end up using for whatever project they are working on.

I actually read this book while I ate breakfast most mornings. Now that's nothing special, but considering I actually continued to read/skim this book even though I have not actively used a Windows box since 2001 should tell you something. This book is clearly written, and does a good job to point out gotchas you might run into through example. But it also does a good job of not overloading you with extraneous info that you could get from other reference sources (every computer book should have something like Appendix C that is nothing more than a list of URLs for reference material). And as an added perk the authors try to be humorous when possible and are even willing to poke fun at Windows.

Sunday, August 30, 2009

Using IronPython to Configure Castle Windsor

Castle Windsor is a popular .NET Inversion of Control container, but it can be complex to configure. A blogger, who's name I can't discover (put your name on your blogs guys!), has written two articles on using IronPython to configure Castle Windsor.
Castle Windsor is a very popular IoC container in the .Net world. Like almost all other containers it can be configured using either a fluent interface or an xml-based configuration file.

The fluent interface has the advantage of being strongly typed, what spares you a lot of errors caused by typos. On the other hand, it is hard coded and can’t be changed easily without recompiling (Actually you could use an IoC container to load your IoC container configuration dynamically but it give a rise to the question: “How do you configure the container to load its own configuration?” )

The other option is to use an xml file. Despite being the most used solution in almost all containers it is really a very ugly solution. The configuration file can get very big and very complicated.

As I am reading IronPython in Action from Manning Publications, I thought I could configure Windsor using Python and a very tiny DSL. IronPython is an interpreted language for .Net framework. It combines the elegance of Python with the strength of .Net. Since it being interpreted it is a suitable solution for configuration.
In the last article I introduced a small Castle Windsor configuration tool using IronPython. This tool enabled us to add service implementation in an easer to read way. On the other hand advanced usages like optional and constructer parameters were not possible.

In this article I will continue developing Pysor (As I called it!) to accept parameters. Before introducing the new functionality I will show what are parameters and when and how would you want to use them. For the sake of demonstration I will borrow the demo application from the very good article series from Simone Busoli about Castle Windsor. If you didn’t read it then go read it all and come back.

Releasing IronPython

Dave Fugate, IronPython tester and the man in charge of releases, gives us an intriguing look into what is involved in releasing a new version of IronPython. Releasing software is always a pain, for any company or project, but even more so for Microsoft.
While it's still fresh on my mind, I'll share a bit of interesting info about IronPython releases. What will blow most peoples' minds is that on average it takes us about five full working days to produce and signoff on a release for CodePlex. For example, we started the release process for IronPython 2.6 Beta 2 on Friday morning and released it today. "Dave, when all you have to do is build the 'release' configuration of the sources already on CodePlex how can it possibly take this long?" you ask. Three words: testing and Microsoft bureaucracy…err I mean "Microsoft processes".
Dave Fugate is also wondering what you want him to blog about...
While I've been waiting for a huge build to finish unzipping on a shiny new Windows 7 test machine, I've been looking over my own and other IronPython team members' blogs. I've come to the conclusion that there are far more comments for posts not dealing with Python, and in many cases the IronPython posts aren't getting any comments at all. The missing interaction isn't necessarily limited to technical posts either from what I've seen.

Is this because we're doing a great job at covering IronPython topic xyz or because xyz just isn't that interesting? If it's the latter, what would you like to see more of and less of?

Saturday, August 29, 2009

Teaching IronPython: D520 weeks 4 & 5

Tony Andrew Meyer is teaching IronPython as course D520 ("Programming") at Northtec, using IronPython in Action as course material. He has been posting weekly updates of his progress including making his course notes available for download.

We've covered his previous entries:
Tony has now posted two more entries, on weeks four and five of the course.
We basically did two things: worked through the MultiDoc example in Chapter 4, and worked on implementing the Airline lab designed in the previous week.

The first recommended reading for the week was Part 1 of Joel Spolsky’s “Talk at Yale”, wherein he tries to relate his study to his career – the part I hoped they would find interesting was the discussion of “geeks” versus “suits”. The second recommended article was Steve Yegge’s “Code’s Worst Enemy”, which is mostly about code bloat. In retrospect, these might not be the best pairing, since Yegge is always long, and this particular Spolsky article is very long (if you read all three parts). However, I was again pleasantly surprised to hear that students were actually reading these.

I was right to set aside a whole ‘lecture’ section of a week for introducing MultiDoc – I used all two hours (with breaks and a late start, actually about 90 minutes) going over the code, and didn’t quite manage to finish – I got up to the very last section, but didn’t have time to go through adding the menubar/menu items (so I did that in the break and showed the working example afterwards).
Chapter 5 of IronPython in Action deals with XML, although it starts out covering some of the more advanced things you can do with functions. I considered skipping this chapter (the function material is perhaps a bit advanced, and covering XML isn’t a necessity), but decided that it was worth learning about XML in .NET (since it’s so common) and that it would make using the MultiDoc example tricky (since the file format is XML) and I really wanted to use MultiDoc. I gave the students notes [PDF], again covering the textbook material that we could look at, the tools (unchanged), key points, and a link to the MultiDoc example code. [...]

I got the feeling that the material on functions was a little advanced, as I suspected. This was, however, the ideal place for me to cover lambdas, as I proposed doing last week, so I added that in. I talked about functions defined in other functions, passing functions as arguments, anonymous functions (lambdas), and decorators. These are important things to learn if you’re going to be a Python developer (or a developer in other languages that let you treat functions in this way), but perhaps belong in a more advanced class. Next year I might skip over this (i.e. assign it as recommended reading), and just rewrite any of the textbook examples to use regular functions. On the other hand, it’s great exam question material!

Sunday, August 23, 2009

A Good Mix 20: Startup time, inline C#, Global.asax in ASP.NET, an interactive shell in the web, Gtk# and more

Another collection of blog entries and projects related to IronPython and the Dynamic Language Runtime.
IronPython 2.0 is significantly slower to start than CPython, which is an issue for those writing command line tools and full applications in IronPython. IronPython 2.6 improves the situation, but as the lament in this short blog entry expresses it is still slow even when compared to IronRuby:
IronRuby 0.9 starts 6 times faster than IronPython 2.6B1 Why is that? Don’t they use the same DLR engine? If IronPython started the same way, probably I would not use ‘CPython plus .Net exposed through MSFT Com’ technique. It is just unfair.
On the usual support channel (Twitter) Dino Viehland explained that the IronPython interpreter does a bunch of work at startup time that IronRuby doesn't. On startup the interpreter imports the site module site just like CPython (and IronPython uses site.py from the Python standard library), this in turn imports a bunch of other stuff - meaning that IronPython is actually doing quite a lot at startup time. They still need to work on startup performance though...This project doesn't have anything to do with IronPython, but it's a really neat idea and would be very easy to adapt for IronPython.

IronRubyInline allows you to put C#, F# and Visual Basic code inline with IronRuby code. It compiles at runtime and allows you to use the compiled classes within the same Ruby code that defines them:
%Q{
public class CsCat
{
public void Greet()
{
Console.WriteLine("Hai u!");
}
}
}.compile

cscat = CsCat.new
cscat.greet # Hai u!
It is similar, although a bit more impressive, to my article on Dynamically Compiling C# from IronPython.
Another lament, this one from the venerable Devhawk, on the interaction between Global.asax in ASP.NET and IronPython.
I always thought of global.asax as an obsolete construct primarily intended to ease migration from classic ASP. After all, ASP.NET has first class support for customizing request handling at various points throughout the execution pipeline via IHttpModule. Handling those events in global.asax always felt vaguely hacky to me.

...

The reason I suddenly care about global.asax is because Application_Start is where ASP.NET MVC apps configure their route table. But if you want to access the Application_Start method in a dynamic language like IronPython, you’re pretty much out of luck. The only way to receive the Application_Start pseudo-event is via a custom HttpApplication class. But you can’t implement your custom HttpApplication in a dynamically typed language like IronPython since it finds the Application_Start method via Reflection. Ugh.
If you’ve ever used Ruby on Rails you’ll be aware of the goodness that is script/console. The console lets you interact with your domain model from a terminal - no need for dedicated GUI screens or tools.

If you can’t see the benefits, then think about this: Have you ever used SQL Server Management Studio to inspect some data in your tables, or even to tweak a value in a column? Yes? Well, imagine being able to inspect items in your domain model, and even create new objects, invoke behaviours (call methods), update properties etc. It’s very handy, and useful for quickly checking or updating data.

I’ve always missed having this feature during .NET development. The good news is that’s it’s quite easy to do using IronPython or IronRuby. My EngineRoom partner, Chris Owen, first got this hooked up using IronPython - so credit to him! Since I’m more familiar with Ruby I’ll use that in this post.
Yet another lament! An intriguing peek into the IronPython development process from Dave Fugate, the IronPython tester extraordinaire:
I've spent way too much time today trying to get a simple _ssl sample to run under CPython to show how broken IronPython's _ssl.cs is. First, I discovered that not only do CPython's tests not work (they're trying to connect to svn.python.org, port 443, which consistently times out); they seem to be disabled as well via test_support.is_resource_enabled('network'). If anyone has encountered the "TypeError: sslwrap() argument 1 must be _socket.socket, not _socketobject" and knows how to workaround it, I'm all ears.
An old blog entry (from 2007) but one I've not seen before. It's an example of using the Gtk user interface from IronPython via Gtk#.
import clr
clr.AddReference('gtk-sharp')
import Gtk
def delete_event (o, args): Gtk.Application.Quit ()

def say_hi (o, args): print "Hello, World!"

Gtk.Application.Init ()
w = Gtk.Window ("Hello, Gtk# World")
w.DeleteEvent += delete_event
b = Gtk.Button ("Say Hello")
b.Clicked += say_hi
w.Add (b)
w.ShowAll ()
Gtk.Application.Run ()
As you can see from the screenshot below, the recipe works with Gtk# and IronPython 2.6 Beta 2 on Windows - however the window is a bit small and unresponsive.


I'd really love to see some articles on using Gtk as a cross-platform user interface with IronPython, Gtk# and Mono / .NET.

Multiple Inheritance in IronPython

Python allows for multiple inheritance, and therefore so does IronPython. With multiple inheritance you provide multiple base classes. As member resolution is done dynamically in Python, when you call a method the class hierarchy is searched to find the correct method. With single inheritance this is straightforward (the inheritance hierarchy is a straight line), with multiple inheritance the method resolution order is used to find methods.

Multiple inheritance can be very useful where you really need it, but it also complicates your code. I believe it should be used rarely, but there are plenty of Python programmers who believe it should never be used. The most common use of multiple inheritance is to provide mixin functionality. It is often used in a similar way to interfaces in languages like C#.

Simon Segal has written a blog entry describing multiple inheritance in IronPython, and how it interacts with inheriting from .NET types:
Question: Does IronPython support Multiple Inheritance / Mixins?

Ans: Yes!

Question: Can I use my C# classes from IronPython (in the IP runtime)?

Ans: Yes!

Question: So, I should be able to use my C# classes in a multiple inheritance scenario from IronPython?

Ans: Yes!
In IronPython you can create Python classes that inherit from .NET types and other Python classes, but you can't simultaneously inherit from multiple .NET types. This allows you to create C# classes that you use as mixins from your IronPython code.

Saturday, August 22, 2009

A Good Mix 19: Reflection, the Silverlight Toolkit, Dynamic JSON, libraries, builds and more

A collection of blog entries and articles on IronPython and the Dynamic Language Runtime from the last few weeks.
Randall Sutton was using reflection to apply rules to an object when he realised he could do it with IronPython, but he was worried about performance. He wrote a quick test to compare the performance and was surprised by the results. The first run was very expensive for IronPython, but from there on in IronPython was much faster than C#.

Without expending any effort actually working it out, my guess is that because IronPython uses reflection so heavily it caches the reflection lookups - and it is this caching that is giving it the performance benefit. Reflection on .NET is not fast, by any stretch of the imagination, and improving the performance of reflection would make a big difference to speeding up DLR languages.
The Silverlight Toolkit is an open source project by Microsoft providing additional controls for the Silverlight browser plugin. By releasing them as an open source add-in it reduces the size of the initial plugin download (at the cost of increasing the size of applications that use them), but also means that they can also be used directly in Moonlight (the Mono version of Silverlight for Linux that went into beta a few days ago) without having to be re-implemented.

The toolkit is divided into five 'bands' offering different ranges of controls. The bands are mature, stable, preview and experimental. The controls in the mature band alone are:
  • AutoCompleteBox
  • Calendar
  • ChildWindow
  • DataGrid
  • DataPager
  • DatePicker
  • GridSplitter
  • HeaderedItemsControl
  • TabControl
  • TreeView
The Silverlight Toolkit Codeplex site has live examples demonstrating the controls. The examples each have three tabs; two showing the XAML and C# for using the controls from Silverlight 2 and 3. The third tab has a dynamic language console that allows you to interact with the control from IronPython or IronRuby. This is a great way to 'try before you buy', to see what they are capable of or use Python's introspection to learn how they work.
This is an intriguing project and I only happened on it by chance when I saw the release appear on the Python Package Index (PyPI - the Python equivalent of Perl's CPAN). The project page says:
Modules for use with IronPython.

The purpose of the modules here are to enhance the IronPython runtime with extra modules for debugging and better integration with the rest of the Pycopia framework.
This doesn't really tell us a lot. The pycopia homepage says:
The Pycopia project is a framework of frameworks for the rapid development of applications related to telecommunications networks, web, data processing, process control, and more.
This is a short blog entry by Simon Segal, the man behind the Entity framework profiler with integrated IronPython scripting. In this blog entry he ponders what to do when there are both Python libraries and .NET libraries available for a task in IronPython. Even with basic things like file I/O and threading you have a choice whether or not to use the standard Python idioms or .NET libraries. They have slightly different semantics of course but it is often not obvious which is a better fit.

Simon also describes his strategy for learning Python and IronPython:
To help myself form my own opinions I decided that the best learning path for me was as follows:
  1. Read the Python 2.5 (CPython) documentations PDF Tutorial and re-type it’s code examples and follow with my own experimenting. This allowed me to get a fairly good overall feeling for the language.
  2. Read IronPython in Action to get a good overall point of relevance to .NET and see how the power of .NET as a platform would impact the Python experience and visa versa.
  3. Read Python in a Nutshell and get a grasp on Python’s roots and how the language is best put to use on it’s original platform (the C implementation).
Some people will surely think that points 2 and 3 should be reversed? Perhaps, but my rationale for taking this approach is that Python is not likely to become a daily instrument in my work and I want to understand as much as possible about it in the .NET world first. I do believe however that to truly understand the power of Python I will need to learn deeply about its roots and oddly I suppose that step in the process will come last.
A blog entry on implementing a way to dynamically query JSON arrays using the Dynamic Language Runtime. The entry creates a wrapper class in F#. It has examples of using the wrapper in IronRuby, using C# 4.0 and the dynamic keyword, and of course with IronPython. The post includes some interesting details on writing dynamic objects implementing IDynamicMetaObjectProvider that can be used from all these different languages. IronPython in particular it seems needs some special treatment because of the way BindGetMember is called instead of BindInvokeMember.
  • Compiling IronPython and IronRuby
The IronPython and IronRuby project files have changed recently, in order to allow them to be compiled by both Visual Studio 2008 and the forthcoming Visual Studio 2010. The details were posted to the IronPython and IronRuby mailing lists:
We have just updated ToolsVersion attribute in .csproj files included in IronPython and IronRuby solutions to "4.0". This change will soon show up in CodePlex and GIT sources.

It allows to open these projects in Visual Studio 2010 (using IronPython4.sln or Ruby4.sln). The binaries produced by both builds (from VS2008 and VS2010) still have dependency only on .NET 2.0 assemblies so there is no change in that.

You'll need Service Pack 1 installed in order to build Ruby.sln or IronPython.sln from VS2008. When you build from VS2008 you'll get a bunch of msbuild warnings:

Project file contains ToolsVersion="4.0", which is not supported by this version of MSBuild. Treating the project as if it had ToolsVersion="3.5".

This is expected, you can ignore them.

Thanks,
Tomas Matousek

IronRuby 0.9: Performance and interop with IronPython

I'm a few weeks behind the curve on this one, but I'm finally catching up with my backlog; IronRuby 0.9 has been released. It looks like they didn't make the version 1.0 by OSCON but they have made a lot of progress.Jimmy Schementi lets us know the highlights of the new release:
Library performance was a big focus for this release; basically going though the Ruby Benchmark suite and making sure any obvious slowness was fixed. As I said in my previous post about OSCON, IronRuby is approximately 2x faster than MRI when running the benchmark suite, but in the near future a complete evaluation of IronRuby’s performance will be done and published on the website.

Antonio Cangiano has already published benchmark results between IronRuby 0.9 and Ruby 1.8.6, and things look really good for IronRuby.

On the compatibility front, the Win32OLE Ruby library is now available in IronRuby. This builds on top of IronRuby’s existing COM interop from version 0.5, letting you script any Windows component/application that exposes a COM interface. Though it hasn’t been fully tested yet, this will make things like Watir work on IronRuby.

Lastly, there have been interop improvements with .NET, most notably making Generic types more friendly to all the crazy things Ruby can do to them, and also with other DLR languages, making it really easy to call IronPython from IronRuby.
Two points particularly worth noting - Antonio Cangiano has posted benchmarks of IronRuby on Windows showing that IronRuby is faster than Ruby 1.9.1 and substantially faster than Ruby 1.8.6. Part of the reason for this could be down to how much worse Ruby performs on Windows than on Linux (which in turn may be down to the old version of gcc it is compiled with on Windows through Mingw - something that is due to change). That aside it is still an impressive result for IronRuby.

The second important point is that IronRuby has matured a lot, to the point where IronPython and IronRuby can now interoperate happily through the Dynamic Language Runtime. Although often portrayed as mortal enemies these languages are really siblings (and which siblings don't argue?) - so a platform on which Python can use Ruby libraries, and vice versa, is very interesting. This has been the vision of the Common Language Runtime for a long time and is a vision shared by the Parrot Virtual Machine, but not yet become a reality.

Because IronPython and IronRuby are both moving targets (developing very fast) it can be hard to try the interoperation in practise, because you need to find a build of both languages that share a common build of the Dynamic Language Runtime. Fortunately the IronRuby binaries for download on Codeplex include compatible IronPython binaries (a build of IronPython 2.6 beta 2) for interop (as a separate download):
Here's a basic example of executing Ruby code from IronPython:
IronPython 2.6 Beta 2 (2.6.0.20) on .NET 2.0.50727.4927
Type "help", "copyright", "credits" or "license" for more information.
>>> import clr
>>> clr.AddReference('IronRuby')
>>> from IronRuby import Ruby
>>>
>>> engine = Ruby.CreateEngine()
>>> source = engine.CreateScriptSourceFromString("puts 'Hello from Ruby'")
>>>
>>> source.Execute()
Hello from Ruby
>>>
Here's a slightly longer example that uses IronRuby to load the Ruby date library and then pulls out the Date class and instantiates it from Python!
import clr
clr.AddReference('IronRuby')

from System import Array

paths = [r'C:\Binaries\IronRuby\lib\IronRuby', r'C:\Binaries\IronRuby\lib\ruby\1.8']
array = Array[str](paths)

source_code = "require 'date'"

from IronRuby import Ruby
engine = Ruby.CreateEngine()
engine.SetSearchPaths(array)
source = engine.CreateScriptSourceFromString(source_code)
source.Execute()

RubyDate = engine.Runtime.Globals.Date

date = RubyDate(2009, 9, 22)
print date.year()

Catching up with Devhawk: Using Python libraries from C#, the HawkCodeBox and a world tour

Devhawk, otherwise known as Harry Pierson, is the program manager for IronPython on the Microsoft dynamic languages team. He is a prolific, even more than me, has a whole bunch of interesting IronPython related posts that I've fallen behind on.

The first set of posts is about building a hybrid Python / C# application, specifically using the awesome Pygments syntax highlighting library from a .NET application.
Harry has been working on a syntax highlighting plugin for the Windows Live Writer desktop blogging tool. He's using Pygments via IronPython 2.6 Beta 2 to do the highlighting, and barring some problems with the import magic Pygments does (known by the IronPython team as “Harry’s Pygments Import Bug”) it works fine with IronPython.
C# 4.0 introduces a new feature, the dynamic type, that will make embedding and interacting with dynamic languages from C# much easier. If you want to use IronPython with .NET versions before 4.0 (which isn't even released yet) you'll have to use alternative techniques.

To create a custom content source for Windows Live Writer you need an on disk assembly with a static type and custom attributes; which means it can't be done directly from IronPython. In this blog entry Harry introduces the posts where he solves this problem.For his hybrid IronPython / C# Windows live writer plugin Harry wants to ship Pygments in the form of a single compiled assembly.
IronPython ships with a script named pyc for compiling Python files into .NET assemblies. However, pyc is pretty much just a wrapper around the clr module CompileModules function. I wrote my own custom script to build the Pygments assembly from the files in a the pygments and pygments_dependencies folders.
The custom script to compile the Pygments package into a single assembly weighs in at only 14 lines of IronPython code. In the rest of the entry Harry discusses the pros and cons of compiling Python packages like this and how he ensures he has included all the dependencies.
In this blog entry Harry writes a generate_html function, in Python, which uses Pygments to do the syntax highlighting. He also goes over how he includes this function in his plugin (as an embedded resource) and sets up the IronPython runtime:
Now that I’ve got Pygments and its dependencies packaged up in an easy-to-distribute assembly, I need to be able to call it from C#. However, if you pop open pygments.dll in Reflector, you’ll notice it’s not exactly intuitive to access. Lots of compiler generated names like pygments$12 and StringIO$64 in a type named DLRCachedCode. Clearly, this code isn’t intended to be used by anything except the IronPython runtime.

So we better create one of those IronPython runtime thingies.
This post is the real meat of the series. Having set up the IronPython runtime he shows the C# for looking up and calling the Python functions.
If you’re working with dynamic languages from C#, the ObjectOperations instance than hangs off the ScriptEngine instance is amazingly useful. Dynamic objects can participate in a powerful but somewhat complex protocol for binding a wide variety of dynamic operation types. The DynamicMetaObject class supports twelve different Bind operations. But the DynamicMetaObject binder methods are designed to be used by language implementors. The ObjectOperations class lets you invoke them fairly easily from a higher level of abstraction.
The rests of the Devhawk posts (yes there are more) cover a range of different topics:
The IronPython project homepage on Codeplex includes a wiki. As the wiki can only be edited by members of the IronPython team, who have a lot on their plate, there is a lot of useful information that could be there that isn't...

This is where you come in. Codeplex now has an "Editor Role" feature and if you are interested in helping you can become a wiki editor for IronPython:
Until now, the only way to give members of the community the ability to edit the wiki also gave permission to edit work items, check in source code and make releases. We’re still working on getting Microsoft at large to understand the benefits of community collaboration aspect in open source, but in the meantime we just can’t give those permissions to people off the team. However, we would love to have contributions to our documentation wiki. With the new Editor Role, we’ll be able to grant wiki editor access without any of the other permissions.
Windows Presentation Foundation (WPF) is the shiny new user interface for Windows (not in Mono unfortunately). One of the touted benefits of WPF over its aged predecessor Windows Forms is extensibility, but unfortunately this doesn't seem to apply to some of the standard controls like the WPF TextBox. Fortunately Ken Johnson has a workaround that he wrote about on the CodeProject.

Harry would like a syntax highlighted REPL (interactive interpreter) for plugging into .NET applications, so he has started to write one - the HawkCodeBox.
Harry is traveling the world, speaking about IronPython and other topics. Here is a summary of his itinerary (so far):
  • Danish University Tour, Sept 7-11
Harry is presenting at four different universities in Copenhagen in four days.
I’ll be visiting Aalborg University, Aarhus University, University of Southern Denmark and University of Copehhagen as well as delivering a TechTalk at the Microsoft Development Center Copenhagen, which is Microsoft’s biggest development center in Europe. I’ll primarily be delivering my Iron Languages introductory talk “Pumping Iron”, but there’s also some interest in language development on the DLR so I’ll be talking on that topic as well.
Not specifically speaking on IronPython, Harry's talk topic is called “Not Everything is a new Nail() : How Languages Influence Design”.
Talks not yet confirmed, but Harry has three talks submitted.
The big annual Python conference.
I finally feel like I might have something interesting to present at PyCon this year. ... We already have one announcement that I think is pretty significant lined up and might have a second depending on how hard I can push LCA and management between now and then. Talk proposals are due October 1st, so any suggestions would be appreciated!
It looks like the folk in Denmark are eagerly awaiting his arrival:
Jeg har inviteret Harry Pierson devhawk.net til Danmark i hele uge 37, det betyder at vi kan komme hele landet rundt!

>>> ironPythonTechTalk = TechTalk()
>>> ironPythonTechTalk.Who
Harry Pierson (devhawk.net)
>>> ironPythonTechTalk.When
Friday, 11th September 2009 09:00 - 11:00
>>> ironPythonTechTalk.Where
Microsoft Development Center, Copenhagen Frydenlunds Alle 6, 2950 Vedbaek
sign-up:
>>> ironPythonTechTalk.WhoIsInvited
You are and did I mention that it's free
>>> ironPythonTechTalk.DoesThisCompile
Ohh yes indeed
>>> ironPythonTechTalk.CantMakeIt
Does these cities and days work better for you!?
{'Aalborg Uni. ': 'Monday, time/schedule: TBD',
'Copenhagen Uni. (DIKU)': 'Thursday, time/schedule: TBD',
'Syddansk Uni.': 'Wednesday, time/schedule: TBD',
'Aarhus Uni.': 'Tuesday, time/schedule: TBD'}
>>> ironPythonTechTalk.StillCantMakeIt
Can\'t help you, sorry...
You can also sign up for his talk at the Microsoft Developer Center in Copenhagen:
Get an introduction to Python on .NET from Harry Pierson (devhawk.net).

The TechTalk will be held in English. It is open for everybody and participation is free.

When: September 11 2009 from 09:00-11:00
Where: Microsoft Development Center Copenhagen, Frydenlunds Allé 6, 2950 Vedbæk

Registration deadline: September 7 or as long as seats are available.

Friday, August 21, 2009

Jeff Hardy: Python compatibility projects and NWSGI release

Jeff Hardy is an extremely prolific developer who has been involved in IronPython for a long time. By testing large Python frameworks and libraries with IronPython he has reported many bugs to the IronPython team.

As part of this Jeff Hardy has accumulated repositories of more than a dozen different projects that he has been porting to IronPython. As you can imagine this is a lot of work and he is offering the opportunity for anyone else interested in these projects to take them on:
I was going through the “Repositories” folder on my machine, looking at all of the stuff I’ve downloaded since, well, the last time I went through this exercise. This time around, a lot of it is stuff I’ve tried to get working on IronPython. Here’s a sampling:
  • Django
  • setuptools
  • Trac
  • Genshi
  • Mercurial
  • SCons
  • CherryPy
  • docutils
  • moin
  • pygments
  • pymarkdown
  • nose
  • sqlalchemy
  • IronRubyMVC
On top of that, there’s the stuff I wrote: NWSGI, IronPython.Zlib, adonet-dbapi, and more.
The good news is that IronPython compatibility with CPython is improving, unfortunately this can actually make it harder to find the root causes of the remaining problems:
Debugging things is also pretty difficult – IronPython is getting good enough that the only bugs are really obscure. It probably takes me ten times longer to find a bug and isolate it into a testcase than it does for Dino to fix it.

Now, with all of that said, I have an offer: if anyone wants to work on the top part of that list (up to and including SCons), I’ll help get you up to speed on some of the issues with those programs, and some of the tricks involved in debugging them (hint: Debugger.Break). This will be limited-time offer, valid until I forget the details of the program in question. I really want to see these things compatible with IronPython, but I just no longer have the time to do it myself.
Jeff isn't backing out completely of course. He's been hard at work on NWGSI. NWSGI is a .NET implementation of the Python WSGI specification for IronPython; implemented as an ASP.NET HttpHandler for use with IIS 6 and 7. The goal is to allow you to serve WSGI web applications with IronPython and the IIS server.

Jeff has just released NWSGI 2.0 Beta 2, which is built against IronPython 2.6 Beta 2:
NWSGI 2.0 Beta 2 is now available. This release in linked against IronPython 2.6 Beta 2, and also fixes a couple of bugs. The big change is that the install does not add NWSGI to the GAC; without IronPython also installed to the GAC, it doesn't make much sense to put NWSGI there.

Thursday, August 20, 2009

IronPython and Silverlight in Turkish

A Turkish blogger named Ibrahim Kivanc, a Microsoft student partner, has posted a series of blog entries showing off some really cool things with IronPython and Silverlight. Unfortunately (for me) the blog is in Turkish, but several of the entries have code to download.

I'm listing the entries here in the order he posted them:
One of the coolest things about Silverlight is its deep zoom technology that allows web applications to zoom in and pan across large, multi-resolution images. This functionality is accessed through the MultiScaleImage class.

This blog entry has XAML and IronPython code for using the MultiScaleImage from IronPython.
Chiron is the tool for developing Silverlight applications with dynamic languages. It is a server that packages apps on the fly and enables you to develop on your local filesystem: edit the Python, refresh the browser and see you changes immediately.
A cool little animation in Silverlight created with XAML and IronPython. It uses a RadialGradientBrush, a Path and various transforms.
Two posts focusing on Python rather than anything specific to IronPython.
Using a Silverlight template for creating a fancy looking user interface with buttons and listbox.

Monday, August 17, 2009

Command Line Options and Configuring Your Engine

The blog entry I'm focusing on here is in Russian. Normally this means I would mention it one of the "good mix" posts but not spend too much time on it, however it makes a useful point that I think is worth explaining:
First of all the author recommends Eclipse 3.5, along with PyDev 1.4.7 and IronPython 2.6 Beta 2 - which in itself is an interesting combination for IronPython development. (Ironic that the big Java IDE has much better support for IronPython than the big .NET IDE.)

One of the new things in IronPython 2.6, and in fact the reason that PyDev is able to have support for things like debugging IronPython code, is support for Python stack frames and APIs like sys._getframe, sys.settrace and so on.

To run IronPython with this enabled you need to use a magic command line switch, of which IronPython has many. You can get a list of them by running ipy.exe -?:


None of the ones above are standard to Python, although some of them would be nice. Useful for the command line are: -X:AutoIndent -X:ColorfulConsole -X:TabCompletion. For access to private .NET members you pass -X:PrivateBinding, for longer CLR exception tracebacks use -X:ExceptionDetail and -X:ShowClrExceptions and to run IronPython in an MTA thread (Multi Threaded Apartment - the default is STA) use -X:MTA.

To enable stack frames you have the choice of -X:Frames and -X:FullFrames. The difference is as follows:
-X:FullFrames promotes all local variables into the heap. So you can always crawl the stack and look/change them for all methods.

-X:Frames only creates the frame objects and if something happens to make us promote local variables (e.g. a closure, or a call to locals(), exec, eval, dir(), vars()) then the local variables will be available for that specific method.
Another useful switch is to enable the profiler built into IronPython 2.6: -X:EnableProfiler. Curt Hagenlocher (core IronPython developer) described the profiler in a blog post a while ago: An IronPython Profiler.

An important question, and the one answered by the Russian blog post, is how do we enable all these features when embedding IronPython instead of launching it from the command line. Fortunately the answer is simple. Create a .NET Dictionary to store options in, and set entries using the same name as the command line switch (the part after the -X:) as the key. Set the value in the dictionary to ScriptingRuntimeHelpers.True.

So, here is the snippet of C# to create a Python engine with the profiler switched on:
var options = new Dictionary();
options["EnableProfiler"] = ScriptingRuntimeHelpers.True;

var engine = Python.CreateEngine(options);
ScriptingRuntimeHelpers lives in the Microsoft.Scripting.Runtime namespace so you will need the appropriate using statement.

To switch frames on as well add this line before creating the engine:
options["FullFrames"] = ScriptingRuntimeHelpers.True;
Easy hey!

IronPython and IronRuby in Debian and Fedora

Microsoft software is not what you might expect in Debian and Fedora / Red Hat (although perhaps it's commonplace now), but this is what C.J. Adams-Collier has been working towards.
I sent an email to the Fedora Legal list asking whether they will accept software released under the MS-PL license. My friend and former colleague, Brett Lentz mentioned that he was concerned that the Fedora folks might not accept software released under the MS-PL. So I asked. I also bcc’d a certain troll on said mail so as to get lots of flame mail. I’m practicing to become a master twitterbaiter.
https://www.redhat.com/archives/fedora-legal-list/2009-August/msg00017.html
In addition, IronRuby/IronPython/DLR may make it in to FC 13, thanks to Brett.
It looks like the upload of the Iron* languages to Debian is imminent. We’ve gotten the debian/watch file downloading a tarball of the git tag we’ve chosen for this release. This was the last bit meebey needed in order to present it to the maintainer of ironpython in lenny. He has agreed to let the Debian CLI Libraries Team adopt ironpython under the conditions that our packaging doesn’t suck and that he remains in the uploaders list.

Now that we have a package suitable for review, we have now presented it for said review.

Let me take a moment to mention how excited I am to be involved in packaging for Debian one of the first pieces of DFSG-compliant software sponsored by Microsoft. This is a true community effort with great work done by a lot of hard-working contributors. I believe that Microsoft has put its money where its mouth is with this project.
As with the great Mono kerfuffle, this hasn't passed entirely without comment.
I was reading a post on Monologue this morning and there was a post getting IronPython & IronRuby into Redhat (I'm assuming the rpm repositories) and it got me thinking about this. My biggest question to this is, WHY?
I posted a response on this blog entry:
Because IronPython and IronRuby both work fine on Mono and are just as useful in environments using Mono as they are in environments using .NET.

Sunday, August 16, 2009

Acceptance Testing .NET Applications with IronPython

The July 2009 issue of Python Magazine is out, and the headline feature is an article on testing .NET applications with IronPython by the honourable Mr. Hartley.



Brandon Craig-Rhodes (editor in chief) describes the article in his blog post on the July issue:
First, IronPython, the .NET version of Python for Windows, is the topic of Jonathan Hartley's article about acceptance testing. He illustrates that, regardless of the language in which you write your .NET application, you can deploy simple strategies to make your application testable through a Python test harness, and thereby bring Python's strong flexibility as a testing language to bear on a product that you might be writing in another .NET language.
The article goes over some of the techniques we use for acceptance testing at Resolver Systems, but applies them to applications written in C# rather than IronPython.

A Good Mix 18: Amazon S3, Professional IronPython, Silverlight Spy, Loading Data from CSV Files and Sharepoint

A collection of projects, articles and blog entries on IronPython from the last few weeks.
Lits3 is "a library written in C# that provides comprehensive and straightforward access to Amazon S3 for .NET developers". The commander is a command line interface to LitS3, written in IronPython by Atif Aziz, and it just had a 1.0 release.
This is a book by John Wiley due out on 20th April 2010. I can't find any other references to it on the net other than this Amazon.co.uk page, but it's been up for a while.
  • Paperback: 504 pages
  • Publisher: John Wiley & Sons (20 April 2010)
  • ISBN-10: 0470548592
  • ISBN-13: 978-0470548592
I've not crossed paths with John Paul Meuller in the IronPython community, but it seems he's written quite a few technical books.
Silverlight Spy is a tool for doing runtime inspection of Silverlight applications:
"Use the built-in browser to navigate to a web page. Silverlight Spy automatically detects the applications embedded in the page and provides various inspection view panes. Explore the XAP package, Isolated Storage, monitor performance, browse the UI visual and logical tree, examine objects, execute code in the DLR shell and more".
As the blurb says, it includes a DLR shell:
"Use the DLR shell to execute dynamic language code such as Ruby and Python right in the Silverlight application's AppDomain. The DLR shell is a very powerful tool that allows you to inspect and modify the application in ways that have not been seen before. The shell comes with a set of samples that should get you started."
Here's a screenshot of the DLR Shell in Action, using IronPython to inspect the live application.
Resolver One uses the excellent LumenWorks Csv Reader library for its CSV file support. This library can also be used from user code and of course directly from IronPython. The reader class is fast and capable, but as with many open source libraries the documentation is somewhat lacking.

This article is part of the Resolver One documentation and covers how to use the Csv Reader from formulae and user code. It is now the best documentation on configuring and using the CsvReader class, from IronPython or any .NET language.
IronPython is great for administering Sharepoint installs, and this snippet of code is one example of how you can use it:
I have been involved in a huge public folder migration project for my company. One particular folder I migrated had a hierarchy of 1500 subfolders. The migration tool was nice enough to migrate the permissions for me for each folder into sharepoint. Cool, right? Yes until my manager told me he just wanted to manage the parent folder in sharepoint and have the subfolders inherit everything. There is nothing in sharepoint out-of-the-box that lets you do this in one shot. Now there are probably many 3rd party sharepoint admin tools that will do this. But here is the poor man's way. Simple IronPython script.

Thursday, August 13, 2009

Scripting the browser with Python - Gestalt

Gestalt is a new project by Microsoft MIX labs that uses Silverlight to allow you to use Python and Ruby to script the browser in the same way that you use Javascript. It is open source and has a home on codeplex:
  • Gestalt.js is a library released by MIX Online Labs that allows you to write Ruby, Python & XAML code in your (X)HTML pages. It enables you to build richer and more powerful web applications by marrying the benefits of expressive languages, modern compilers, AJAX & RIAs with the write » save » refresh development model of the web.
  • Gestalt is a way of building more powerful web applications without changing the way you work. Learn more
  • Gestalt is built on IronRuby and IronPython for Silverlight
  • This CodePlex project makes available the sample and source code for Gestalt.
Gestalt is still young but Steve Gilham has been trying out the beta:
The Python-in-the-page concept for Mix's Gestalt is great -- but as a beta still has some rough spots. This is the result of some initial playing...

Recompressing the .xap file at high compression will get you around ~25% reduction in size (for the python .xap, at least). The extra 5k saving for minifying the loader script is small change by comparison.

External script files () appear to be not (yet?) supported -- I get an unhandled error when I split the python code for the 02_python.html sample out.

The round-the-houses Invoke() style needed to call into DOM properties is a bit tedious (as seen in the canvas sample).

There do appear to be some subtle interactions with existing scripts -- adding a hello-world button into an existing page with scripts+canvas+... silently did nothing for me -- I've not yet stripped this down to a simple problem case, but do be wary of this.

Teaching IronPython: D520 weeks 2 & 3

Tony Andrew Meyer has now been teaching course D520 ("Programming") at Northtec, using IronPython in Action as course material, for three weeks now.

He's been writing up how its going and posting the course notes online (week one described here), and weeks two and three are now up:
Here’s my material from the second week of “D520: Programming” (in IronPython). The students got some brief notes [PDF] and the first proper lab exercise [PDF]. The recommended reading this week was a post by Lukas Mathis about poor hardware design (and lessons to be learnt), and a post by Wil Shipley about tracking down a Delicious Library bug. The notes are again in four sections: textbook chapters (this week chapter 3, which is fairly essential reading), tools (same as last week, although I also recommended IronPython 2.6b2), key points, and example code (from chapter 3 of the textbook).
When planning the semester’s schedule for D520, I choose a few topics that seemed large and gave them a two-week time-slot. One of these was chapter 4 of IronPython in Action, which covers duck typing, design patterns, and introduces the MultiDoc example that’s used throughout the middle section of the book. One of the concepts that the course has always (at least, as long as I have known it) tried to push is the importance of design – not just user-interface design (although that’s both important and covered), but the importance of doing at least some planning before starting to write large amounts of code. In the last couple of years, I’ve moved the course away from focusing on extensive formal design to also cover design patterns and testing (particularly automated testing, like unit tests). Since this is such a major issue for the course, and since I planned on using MultiDoc as an example in class (I try to always have an example that continues on from week to week), this seemed like an obvious point for a two-week session.