Wednesday, September 30, 2009

O2: 'Open Platform for automating application security knowledge and workflows'

Dinis Cruz is a Web Application Security expert and is the chief technology evangelist for the Open Web Application Security Project (OWASP).

Dinis has been blogging about an "open source project designed to improve the productivity and capabilities of security consultants who perform application security engagements".

This project support Python on the JVM through Jython and Python on .NET through IronPython.
 In a nutshell, O2 is a bunch of (about 25) open source modules/tools that help with the multiple aspects of performing application security engagement (in most cases by extending the capabilities of a several Commercial and Open Source tools).

There is a large number of O2 modules that are designed to work specifically with the Ounce 6.x product (Ounce Labs Static Analysis engine), and several other O2 modules which are 100% independent and can be used using only freely available or Open Source tools.

One of the most powerful features of O2 is its scripting and customization capabilities. Currently O2 supports scripting in
any .Net language (with an O2 module dedicated for coding and debugging C#),
  • Java using IKVM
  • Python & Java with a via Jython and
  • Python & .NET via IronPython.
Everything in O2 is exposed via powerful object models and schemas (which are designed to make the security consultant much more productive).
The OUNCE O2 project has a page and code samples on using Python / C#.
I recieved last week a great set of OSA/O2 questions which are better answered here (see also the code samples at the end which are a good examples of the powerful O2 Finding's filtering capabilities).

...


Using the O2_Tool_Python to write a Python script:

step 1) Open the O2 'Scripts' module (from the 'Loaded O2 Module' menu)
step 2) select a *.py script from the samples on the left hand side
(optional) step 3) save it with a different name (use the properties button to open the save dialog)
step 4) enter the python script below in the source code area (you will need to fix the path to your local webgoat.ozasmt file and O2_Binaries folder)
step 5) select 'IronPython' engine
step 6) click on 'Execute on External Script Engine' (the big red exclamation mark). This will save and execute the code.

Configuration with IronPython

Mark Bloodworth is an architect evangelist who has blogged several times about IronPython. He has put up two more blog entries on using IronPython as a configuration engine.
Configuring applications in code seems like a good idea.  Configuring them in code written in a dynamic language seems even better.  Let’s look at a simple example with IronPython.

...

What next?
I’ve kept this example as simple as possible to show the principles.  The advantages of using Python code (it would be simple to use IronRuby, too) are that it’s simpler to write, testable, more powerful and doesn’t need pre-compiling.  And it could do much more than set a couple of properties.  In C# 4.0 the dynamic keyword would allow the Configuration class (and classes like it) to be dynamic, so that the properties and methods could be referenced at runtime without having to be declared ahead of time.
Recently I posted a short article about how to do simple configuration with IronPython.  I figured that it would be easier with .NET 4.0 thanks to the dynamic support.  And it is.  Using Visual Studio 2010, create a new Console Application.
...
The result is (I think) more simple and elegant than the configuration code to which I’ve become accustomed.

Monday, September 28, 2009

New Release of VisioAutomation library

Microsoft Visio is a popular vector diagramming programming for Windows. As with other Microsoft programs it can be automated and extended with COM, which can easily be done from .NET languages like C# and IronPython.

VisioAutomation is "a .NET Library that simplifies automating Visio from .NET languages such as C#, F#, IronPython and a set of tools that use that library." It is useful for:
  • Writing managed-code Add-Ins for Visio 2007
    Writing tools to automate Visio 2007 from command line
    Use it as a reference to learn how to do common Visio automation tasks
Version 2 of VisioAutomation has just been released. As well as various tools and libraries the latest version includes an experimental interactive environment built on IronPython and libraries to support using VisioAutomation from IronPython.
Today I’m releasing the latest version of the VisioAutomation library on CodePlex: http://visioautomation.codeplex.com/

This was a very large update and took about 6 months of effort.

CHANGES SINCE 1.X
  • The library itself has been reorganized into several distinct pieces (each in a distinct namespace)
    • VisioAutomation – the core low-level library – I don’t expect people to spend much time using this directly
    • VisioAutomation.Scripting – if you are writing interactive tools that work with Visio then this is the part of the library that will make using controlling Visio much easier
    • VisioAutomation.DOM – This one is a document-oriented API to control Visio – consider it low-level, you generally don’t need to interact with it
  • The tools have been reorganized
    • Visio Power Tools – this provides some extra features to Visio (such as Exporting to XAML) and the ability to create really nice gradients
    • VisioInteractive - (Experimental) A library meant for use by an interactive IronPython 2.0 session
    • VisioPS - (Experimental) an Module for IronPython 2.0 that lets you control Visio from the command line

FUTURE PLANS
  • VisioPS will be pulled into its related project I am working on – I haven’t finished with the concept but essentially think of it is a “simple business intelligence & analytics from the command-line” – the idea is to make it easy to play and render human-scale data
  • VisioAutomation – evolutionary changes
  • VisioAutomation.XML – A library to create Visio .VDX files directly from C# without having Visio installed (yes, it’s possible and it’s not even that hard)

PARTING THOUGHTS
Even if you aren’t interested in using the code for the library, check out the Visio Power Tools add-in – it adds some nice end-user features.

A Simple IronPython ActionFilter for ASP.NET MVC

ASP.NET MVC is the all new web framework for .NET inspired by frameworks like Rails and Django. There are a few examples of IronPython and IronRuby integration with ASP.NET MVC but the dynamic languages support is incomplete. Harry Pierson has blogged about at least one of the problems.

Despite this there are various places where IronPython can be integrated with ASP.NET MVC, as demonstrated by a new article (with example code for download) on Code Voyeur:
There are times when an action filter has transient logic that will change frequently. Logging, authorization or request sanitizing rules are examples of filters that may require extensive configuration to achieve needed flexibility.

Rather than create several filters that make excessive use of appSettings, it is possible to use the Dynamic Language Runtime and script files to achieve the same result. This article will demonstrate how to create a filter that uses a simple convention to map action filter hooks to functions in IronPython script files. Like other CodeVoyeur articles, this one assumes the choice of a statically typed language for core application logic has already been made.

The PyFilter class will be the core file that performs the execution of the IronPython code. It will override each of the four virtual action filter methods from the ActionFilterAttribute base class.

Sunday, September 27, 2009

Darrell Hawley on IronPython and LINQ

Darrell Hawley is a Microsoft MVP with an interest in IronPython. We've featured some of his previous writings here, here and here.

He's made a couple of posts on using LINQ from IronPython. LINQ is a .NET feature that give first class syntax in C# to queries. Under the hood LINQ is implemented as a series of extension methods and classes. In it's basic form LINQ is similar to Python generator expressions. As well as working on objects LINQ expressions can be used to generate SQL queries. A Python equivalent would be the Dejavu project.

LINQ is a language feature where multi-line lambdas really show their power; something that isn't possible in Python of course where lambda functions are restricted to a single expression. The power of LINQ causes Darrell to describe it as:
Python developers would be better served to think of LINQ as wrappers around the map, filter and reduce functions. In fact, LINQ is not much different than the “recipes” found in the official itertools documentation.

The first of Darrell's blog entries on the topics is actually a correction to a comment about LINQ Darrell made in a Devlink presentation. He follows it up with a more general look at what LINQ is capable of and how to use it from IronPython - either using the LINQ classes or achieving similar effects with Python libraries.
 You have to remember that IronPython does NOT support extension methods and since LINQ is pretty much a collection of extensions methods, there’s a problem. Why this design decision? Because in order to support extension methods, the IronPython team would have had to make changes to the core language which would have, of course, been rather un-Pythonic. This means that the code for consuming LINQ is going to be somewhat messy. The following snippet gives an example of consuming the Average function of the Enumerable object.

clr.AddReferenceToFileAndPath(r"C:\folder\System.Core.dll")
from System.Linq import Enumerable
from System import Func

list = [1,2,3,4,5]

print Enumerable.Average[object](
    list, Func[object, int](lambda x:x))
So the question still stands: does IronPython need LINQ? In my last post, I described how to use LINQ from an IronPython application and implied that I didn’t think it was the best idea. Don’t get me wrong, I really enjoy writing LINQ statements, I just don’t think LINQ fits well with IronPython. Why not? The first problem I have with the IronPython/LINQ combination is put on prominent display in this code sample.

print Enumerable.Average[object](
    list, Func[object, int](lambda x:x))

If you are a static language aficionado, you probably are wondering what the problem is. If you’re a Python developer, however, the fact that I’m explicitly naming the “object” and “int” types is sending up giant, red flags. Python folks don’t really care what type of object it is they’re using, they only want to know what the object can do and what it contains (i.e., duck-typing). It’s much easier – and more readable - to simply write a completely Python version of the Average function.

A Good Mix 23: IronPython 2.0.3, Eclipse 3.5, a WPF Hyperlink, dynamic in C#, MathNet and more...

Another collection of blog entries and articles related to IronPython and the Dynamic Language Runtime.
With IronPython 2.6 Release Candidate 1 out of the door you might think that IronPython 2.0 was already defunct. Not true! David DiCato, an IronPython core developer, just posted to the IronPython mailing list asking what bugs people would like to see fixed in a 2.0.3 release:
As we work towards our IronPython 2.0.3 bugfix release, Dino and I would like to get a feel for which bugs left unresolved in 2.0.2 are most important for us to fix in the next release. Please let us know ASAP if there’s an issue you’d like to see fixed in IronPython 2.0.3. Thanks!
 A blog entry (in German with just a hint of Latin) from Rainer Schuster about how he bent Eclipse 3.5 (with PyDev) to his will for IronPython development. Google translate reveals his conclusion:
From now on you is CodeCompletition and much more are available. Separate assemblies that are not from the framework can use Project-> Properties-> PyDev - PYTHONPATH be added. A good overview of PyDev functionality can be found on the homepage under PyDev Features.

I hope Micorosoft not slept through it and gets with Visual Studio as IDE for IronPython back the crown. As so often has once again come to show someone from the Open Source camp and how it works. The funny thing is that it happens even at this time Microsoft's own open source project, which also holds in more formal entry into the framework. JAVA strikes back ;-)
This entry is in Japanese but the substance of the blog entry is code. The entry shows using the Hyperlink class from the Windows Presentation Foundation (WPF) UI to create an about dialog.

The code is used by SeeMe, an open source Python application for Windows and Linux.
Chris Burrows is Microsoft developer who has blogged extensively about the new dynamic keyword in C# 4. dynamic is a mechanism for dynamic typing (and duck typing) in C# made possible by the Dynamic Language Runtime. Introducing dynamic elements into a static type system is not without its complexities and Chris does a great job of going through the implications and details.

In his latest blog entry on the subject Chris discusses using dynamic with type constraints:
You cannot specify type parameter constraints that involve dynamic. There are a few reasons why we disallow this. The first reason (and possibly the least satisfying) is that we have nowhere good to put the metadata that says the List<object> in question is really a List<dynamic>. Well, not without concocting some scheme to associate the DynamicAttribute with constraints that appear on methods and classes.
If you are interested in using dynamic in C# then Chris' blog will be a useful resource. If you program in a dynamically typed language then you can just marvel at all the complexity you are saved from.
This is a project that has been around for a while but that I have only just noticed.

Math.NET is an open source mathematical toolkit written in C# for .NET and Mono:
Math.NET aims to provide a self-contained clean framework for both numerical scientific and symbolic algebraic computations. The project is covered under the GPL or LGPL, vary among various components.
 Iridium is the numerical foundation of Math.NET, aiming to provide commonly used mathematical elements for scientific numerical computations. It offers the infrastructure for linear algebra, random generators and distribution, special functions, FFT and more.
 The codeplex project IronPythonMathNet is a reference document (with examples) for using MathNet.Iridium from IronPython.
We’ve been giving a lot of thought to our choice in Programming Language for Journey of Souls. For a while, it was starting to look like we were going to end up writing everything in PHP or Python. I think we’ve made our decision (or at least are very close). It ended up coming down to a few things that PHP and Python just don’t have any out of the box support for. Things that it would be extremely stupid for us to roll our own implementations, when another language offers the feature up really well. The first of these being Events and Event Handling. The second being a way to write game scripts in a way that they can be replaced on the fly while the game is still running. Here’s some of the details from our research and decisions.

...

That solved one of our two problems, but C# is a compiled language. If we need to make changes to the game, that’s going to require a recompiling and stopping the game to drop in the new binaries and … wait a minute. I remembered the DLR. Not only did I remember the DLR, I remembered that the languages (IronPython, specifically) can be embedded. Meaning, we can do our scripting for the game in IronPython, and it offers us up the opportunity to do nifty things like reload game scripts at runtime. This is absolutely amazing because it minimizes downtime for the game, in the long run, and allows us to immediately add new features and such to the game without a restart. It also still lets us do a lot of stuff in one of our preferred languages, while still being able to utilize C# for any heavy lifting that needs to be done.
In my mind, this seems like a very win-win scenario. We’re going to be future-looking, so I’ve already been considering the use of things like C# 4.0’s dynamic keyword for making interacting with IronPython that much easier. It also helps that I’ve already been using the Visual Studio 2010 and .NET 4.0 Betas. We still have a long way to go before we get something usable, but locking down our language choice is a huge milestone and an issue that’s been bothering us for a long time now. With that gorilla off of our back, we can start working more on the actual design of the game and how we want the core pieces to interact, knowing that we have an Event model that’s built-in to the language we’ll be developing in.
I’ll keep the updates coming as we have more things to share.

Tuesday, September 22, 2009

IronPython 2.6 Release Candidate 1 Released

IronPython 2.6 is a version of IronPython targeting compatibility with Python 2.6. As well as the new language features introduced in Python 2.6 there are a whole host of new features specific to IronPython.

The big new features in IronPython 2.6 are:
  • The __clrtype__ metaclass for data binding and .NET attribute support
  • Implementation of the ctypes module
  • Support for Python stack frames and sys.settrace, which means the pdb debugger works
  • Better performance through adaptive compilation
  • Faster startup
The announcement of Release Candidate 1 from the IronPython team:
We’re pleased to announce the release of IronPython 2.6 Release Candidate 1 which can be freely downloaded at the codeplex site.

Over the development span of IronPython 2.6, exactly 417 bugs have been fixed.  Since the release of Beta 2, we’ve addressed the following:
  • Non-hosting related APIs previously found in Microsoft.Scripting.dll have been migrated to Microsoft.Dynamic.dll.  The rationale behind this decision is that we’re aiming for compatible DLR binaries with the next major release of IronRuby, namely 1.0.  Microsoft.Dynamic.dll will likely end up being a bit different between IronPython 2.6 and IronRuby 1.0, but we intend on keeping Microsoft.Scripting.dll and Microsoft.Scripting.Core.dll exactly the same.  By doing this you’ll be able to utilize the DLR hosting APIs to run both IronPython 2.6 and IronRuby 1.0 code from the same .NET application!
  • Documentation distributed with the release has been updated
  • A plethora of bugs have been fixed over the past two months with special emphasis given to CodePlex work items with lots of votes
If no major issues with this release candidate are discovered, we hope to ship the final 2.6 release in a little under a month.  Anyone planning on upgrading to 2.6 should try out this release candidate and let us know of any issues you find ASAP.

Sunday, September 20, 2009

A Spinning Solar System: 3D Graphics in IronPython and Resolver One

Back in early 2008 I posted a short entry on using IronPython with OpenGL and the Tao Framework for 3D graphics.

Tao is a set of bindings to OpenGL and SDL for cross-platform "media application development" with .NET and Mono.

My boss at Resolver Systems, Giles Thomas, has been having fun using Tao inside Resolver One. Resolver One is a "Windows-based spreadsheet that integrates IronPython deeply into its recalculation loop, making the models you build more reliable and more maintainable". Because it is so easy to program with IronPython, my Giles decided it would be a good platform for experimenting with 3D graphics - including a full animated version of the solar system with the paths controlled from the spreadsheet. Both entries have a video of the results and IronPython code to download.
I’ve been playing around with 3D graphics recently, and decided to find out what could be done using .NET from inside Resolver One. (If you haven’t hear of Resolver One, it’s a spreadsheet made by the company I work for — think of it as Excel on steroids :-)

Once you’ve loaded up the spreadsheet, a window will appear showing the spinning cube with the Resolver Systems logo that appears in the video. In the spreadsheet itself, the worksheet “Path” contains the list of 2,000 points that the cube follows while it spins; as in the video, these are initially all (0, 0, 0), so the cube just sits in the centre of the window spinning.
The entry has a video of the result, a walkthrough of the code, and shows how to control the path of the spinning cube using column-level formulae. Part two takes the concept further:
In my last post about animated 3D graphics in Resolver One (the souped-up spreadsheet the company I work for makes), I showed a bouncing, spinning cube controlled by the numbers in a worksheet. Here’s something more sophisticated: a 3D model of the planets in our solar system, also know as an orrery.

A window showing the 3D animation of the solar system will appear while the file loads. To manipulate it, drag the mouse left/right to spin the solar system around the Y axis, or up/down to spin around the X axis (left to right in the initial state). Zoom in and out using the mouse wheel or +/-.

Here’s how it works in more detail; I’ll start with the spreadsheet and then go on to the OpenGL/Tao/IronPython support code that lives in OpenGLPlanets.py later.
 This isn't the only "innovative" (read crazy) use of Resolver One I've seen recently. Here's another that uses the spreadsheet as a Twitter client:
I have developed a little Resolver One spreadsheet that can update your tweeter account, get your friend’s list or get the list of those who are following you.

Blaze: Automated Desktop Experience

According to its homepage, "Blaze is an innovative utility designed to automate most of the unique recurrent tasks that arise from everyday usage. Such tasks can be launching applications, making small calculations, sending emails, inserting recurrent excerpts of text or even renaming files!"

Blaze is Windows only and embeds IronPython for user scripting. It is open source, and the latest release is from Friday (September 18th) so it is under active development. Blaze is available as an installer or a portable version that can be used without installing.
There isn't yet a manual for developers. However, it's pretty easy to extend Blazes functionality with IronPython. Take a look on Scripts Folder. There are already two scripts build in which you can get inspiration from.

To use the ContextLib, there is an outadated .chm file in Blaze's instalaction directory, in the "Docs" folder. It should work just fine for most common features. In the future I will update this documentation properly.
Not only can you write new automator scripts in IronPython, but macros you record will be saved as IronPython scripts:
Blaze has the ability to record Adaptive Macros, which are macros that can adapt to new window positions. These macros also have the advantage of being recorded to IronPython files (.py), making them really easy to edit with notepad.
Not so sure about the advice to use Notepad for editing Python files though...

Saturday, September 19, 2009

Using IronPython to Configure Castle Windsor

This is part 3 of a series of articles, by Moukarram Kabbash, on configuring the Castle Windsor Inversion of Control container for .NET with IronPython. We previously covered the first two articles in the series.
In the first two articles I introduced Pysor, the Castle Windsor configuration tool using IronPython. Now I have added some exciting functions to exploit the nice hash table and list syntax feature in IronPython.

Since the second part of this series is possible to add (named) parameters to component registration. It accepted only both literal and referential scalars. In the current revision you can exploit the nice list syntax of Python using the square brackets to add arrays and list parameters.
The configuration project is called Pysor and is available for download from Github:
The updated to-do list is now
  • Adding a nicer API for referencing assemblies and importing namespaces (I have now idea how to do it).
  • Adding parameters to be passed to the constructor.
  • Passing parameters as lists or arrays.
  • Dictionary based parameters
  • Referencing already registered implementation inside the same configuration script.
  • Documenting and signing the assembly
  • Lifestyle management
  • Considering turning Pysor into an Interpreter to be used as the built-in XmlInterpreter
All remarks , ideas, bug reports, etc. will be appreciated.

A Good Mix 22: Win a book, Mandlebrot Performance, Double clicking in Silverlight and more...

Another collection of blog entries and articles related to IronPython and the Dynamic Language Runtime.
Allan Juhl Petersen has a competition running until 29th September where you can win a book on IronPython (and a good book it is too) or F#:
I have created 7 questions, in different programming languages such as C#, F#, Ironpython, C++, Java and of course the well known and liked true or false. The questions are designed to give a number of points, where the maximum number of points given can be read in the text of the assignment. The rules are simple, the one with the highest score on points, will win. Points will be given for correctness and effort. An assignment might give 6 point, but if not entirely correct, it will be given some points for effort.
The winner gets Office 2007 Student version and gets to choose between two books ? Foundations of F# or IronPython in Action - and the runner-up gets the one that the winner did not choose.
This article will compare the latest incarnations of Ruby, with the latest in Python, Groovy, PHP, Lua, Perl and Java too, to have a comparison with a pre-compiled language. We will see, how scripting languages behave if applied to fractal geometry, more precisely an family Mandelbrot algorithm.

Browsing on the net, I found a comparison very interesting but a bit dated, dates back more than two years ago. Since then things have changed and I took advantage to make an update, not including all of those languages but only for more known. This is an opportunity to compare Ruby and Python versions even in their Java and. NET, an intention that I had since long time.

These are the performance results obtained from an average of five runs, took after have executed some void attempts (I have not trusted the VM startup):

Language      Time (in seconds)  slower than java
_____________________________________________________________
Java 6 update 15    0,153
Lua 5.1.4           0,815              5x
Php 5.3.0           2,083              14x
Python 2.6.2        2,269               15x
Python 3.1.1        1,566               10x
Jython 2.5.0        2,850               19x
Jruby 1.3.1         2,466               16x
Groovy 1.6.3        6,491               42x
Ruby 1.9.1 p129        2,688            18x
Ruby 1.8.6 p368        6,863            45x
Ruby 1.8.6 p111        9,709            63x
IronRuby 0.9.0        6,038             39x
IronPython 2.0.2     0,978              6x
Perl 5.10.0         2,722               18x

As you can see, for this type of calculation IronPython comes out faster than Ruby, IronRuby and CPython.
I have recently been wondering about how to build a generic system for parsing and processing lots of EDI messages in such a way that a minimum of work is needed when a new message type is to be processed by the system. The syntactic format of EDI messages is fairly consistent, but the semantics of particular fields are open for interpretation. Thus, I thought it should be possibly to write a general parser for turning a message into a tree structure (much like an abstract syntax tree for “the EDI language”) which would be appropriate as input to the processing phase of the system.

Having recently picked up “IronPython in Action” I decided to apply IronPython to the task. Since this was only an experiment, I decided to somewhat give up the idea of a tree structure and instead leverage the dynamic programming capabilities of IronPython. Given an EDI message at runtime I wanted to generate an object with properties corresponding to the segments, subsegments and elements of the message.
A new release for this mysterious project on the Japanese sourceforge site. The main project page has more details - but in Japanese.
Silverlight has a left click event for controls, but not a double click event. This makes for fun when implementing some user interfaces. The Evil Monkey Labs blog has an example of how to work around this in IronPython:
Silverlight rather oddly lacks a double click event. You can detect single click, but for double clicking you're on your own. I found some examples for C#, but none for IronPython and had, a few months ago, ported some code I found for IronPython.

I've been using it in application development and user testing for about 6 months and it works fairly well in both Silverlight 2 and 3, although YMMV.

Monday, September 14, 2009

IronPython in Action Reviews: Dror Helper, Benjamin Peterson and Life in the Cove

IronPython in Action is the first book on IronPython and is aimed at both .NET developers new to Python and Python developers new to .NET.

Three more reviews have been posted, one by a core-Python developer (the release manager no less), one by a .NET developer and the third, well, hmm...
Benjamin Peterson is the Python release manager and a contributor to the PyPy project. You might say he knows a bit about Python. Here's an extract from the review:
IronPython in Action seems to do a very job, overall, of catering both Python programmers tiptoeing into IronPython and .NET and C# developers finding the light of dynamic programming.

I found the web programming part of the book, especially the part on Silverlight, most interesting, since embedding Python in the browser seems like a lot more fun than writing cross-browser JavaScript.

Before reading this book, I had dismissed .NET as a non-cross-platform hunk of Javaish APIs. I see now, though, that IronPython is able to combine the beauty of Python with some of .NET's better APIs (I would still rather use PyQt for GUI programming. Windows Forms has not improved.) to make a powerful development platform.
Dror Helper is a .NET developer who has been exploring development with IronPython and made a series of posts about his experiences. Here is an extract from his review:
One of the important aspects of learning new language is learning on the tools needed for development, I was pleasantly surprised to find that the authors didn’t forget to show the reader the available IDEs and libraries that he can use for development along with small tutorials and thoughts on their usage.

The book is a good read and manages to transfer the authors (Michael Foord and Christian Muirhead) experience and knowledge on IronPython.

So if you’re a .NET developer that wants to learn what dynamic programming is all about or a CPython developer looking for a way to enter the .NET world you really want to read this book
Programming can be an esoteric art at times, but even I was surprised by the intro to this blog entry:
Have I written my review of the book, "IronPython in Action"? I don't remember. Not that I want to, giving away all the secrets tucked away in the book that led my buddies to discover the truth, the hidden language, the 3D helix that governs the activities of this sector of the universe and leads to temporary vortices we call ourselves.
I don't guarantee enlightenment if you read IronPython in Action, but it good be an important step on your path...

More from Mono: Moonlight 2, Monodevelop 2.2 and Debian Packages

Several of the recent entries here have been about Silverlight and IDEs. We continue the theme in this entry, but looking at things from the Mono side of the fence.
Silverlight is a cross-platform, cross browser plugin by Microsoft that allows you to execute Python code in the browser with IronPython. It is a Rich Internet Application framework with a user interface model based on WPF (Windows Presentation Foundation) and full access to the browser DOM.

Unfortunately cross-platform only means Windows and Mac OS X, leaving Linux users out in the cold. Fortunately the Mono team have stepped up to the plate. Their implementation is called Moonlight, and Moonlight 2.0 (the version that can run Python code) has recently reached beta. It is a Firefox plugin that runs on Linux and is capable of running most of my IronPython Silverlight examples.
Moonlight isn't the only interesting beta to have come out of the Mono household in recent days; MonoDevelop has also reached an exciting new milestone with the release of 2.2 Beta 1.

MonoDevelop is the Mono IDE, originally created as a fork of an early version of SharpDevelop (a Windows only open-source .NET IDE).

The important new feature of MonoDevelop 2.2 is full cross-platform support, including Mac OS X and Windows.

Mono 2.2 also adds official support for Python development:
The Python Binding has been greatly improved, and we are releasing it as an official MonoDevelop add-in for the first time. Here is a summary of the features supported by the add-in:
  • Robust and fast code completion support
  • Quick class/method finder at the top of the file
  • On-the-fly syntax checker, with support for Pyflakes.
  • Support for Python 2.5 and 2.6
  • Code folding for Python blocks
This is not the same as IronPython support, something that is still on the MonoDevelop wishlist. The Python support is for CPython, but the good news is that IronPython support would be able to share a lot of features from the Python bindings - all we need is someone to step up and provide an IronPython binding...
If you're interested in trying out IronPython with Mono then these preview packages may be of interest... The recent versions of DLR languages need very recent versions of Mono. apt-get install ironpython works, but currently installs a very old version of IronPython (1.0 I think). These packages will enable the most recent versions to run.

The Mono packages are maintained at: Mono for Debian.

Sunday, September 13, 2009

IronPython Tools and IDEs (and important news about PyDev)

A frequent question on the IronPython mailing list is "what IDE should I use with IronPython?". For many .NET developers the question is phrased slightly differently, "how do I use IronPython in Visual Studio?".

Just as with Python this question is not easy to answer, depending to a large extent on what IDEs you have used before and personal taste. It is further complicated in IronPython with the need for features from a .NET IDE and from a Python IDE in order to fully use the language and programming environment.

Popular options include IronPython Studio (either integrated with Visual Studio or standalone), Visual Studio itself, SharpDevelop, Eclipse with the PyDev extensions and the Wing IDE from Wingware.

All of these tools have different levels of support for working with Python, the .NET framework, and IronPython specifically. This article looks at all of these IDEs and their features. It also covers other editors, plus common tools for Python development (code quality, refactoring, debugging, profiling, complexity and so on).
One of the IDEs that comes out very well is Eclipse with PyDev - yes, the major Java IDE has better support for IronPython development than the major .NET IDE!

When I first wrote the article, a whole two or three weeks ago, the PyDev extensions that add IronPython support to Eclipse were a commercial product (costing all of $42.50, but still commercial).

Since then PyDev 1.5.0 has been released and Aptana, the company that owns them, have made them fully open source:
Today, Aptana is proud to announce that Pydev and Pydev Extensions have become a single plugin, with all the available contents open source (and freely available for anyone) in the 1.5.0 release (it's the same as 1.4.8 but with all the code open source).
Here are the summaries from the major IDEs reviewed:

IronPython Studio and Visual Studio

Visual Studio is the 400 pound gorilla in the world of Windows IDEs. It is a great shame that its IronPython support is so poor. IronPython Studio is not a production quality IDE and is awkward to use even when integrated into Visual Studio. For anything other than small Python projects using Visual Studio as your main Python IDE will be a painful experience. The IronPython team are aware of this and are working on changing the situation!

It isn't all bad news however. Debugging IronPython code in Visual Studio is possible, and IronPython Studio helps with this if you are able to use it. Using C# in IronPython projects is so simple that there is no need to have designers that directly generate IronPython code. In addition, although not suitable for large projects, if all you are doing is including small amounts of Python in a mainly C# project then you will probably find using IronPython Studio bearable.

SharpDevelop

Overall excellent support for IronPython, especially when compared to Visual Studio. SharpDevelop does however fall down when compared to more specific Python IDEs; lacking features like intellisense, calltips and even Python oriented auto-indent. These features are ones that I consider essential for general development but I will be keeping a careful eye on how SharpDevelop progresses.

Even though I don't use it as my main IDE I am keeping it installed so I can use the debugger when I need it.

Wing IDE

Unlike the IDEs we've been looking at so far, Wing is a great Python IDE. Wing has the advantage of being cross-platform and can be used for working with IronPython and Mono on Mac OS X or Linux. Wing is also useful for both CPython and IronPython.

The scripting API is one of the killer features of Wing, it is very easy to extend (in Python of course) and integrate new tools. The intellisense and source assistant are its other killer feature and I find the Subversion integration, code browser and project browsers invaluable too. Unlike Visual Studio Wing isn't opinionated about how your project should be laid out, which makes it less of a pain to use with existing code.

By being a Python IDE rather than a .NET IDE there are features missing, especially designer and debugger support, and although the intellisense is very good it requires running an external tool on libraries you are using with IronPython. As few of the other IDEs have this feature at all, and there are other ways of getting a designer (I use Visual Studio Express generating C# for this) and debugger, Wing is my preferred IDE for IronPython development.

Eclipse and PyDev

In many ways Eclipse and PyDev has the most powerful IronPython integration of all the IDEs we have looked at. It is a capable Python editor and the added features for IronPython are built very naturally on top of this. If I wasn't so familiar with Wing I might be tempted to switch. Scripting Eclipse is more complex than Wing, and it is more opinionated about things like project layout, but having an IronPython shell built-in is a nice feature.

It is at least slightly ironic that the largest Java IDE has better support for developing in a .NET language than Visual Studio.

Intersection of built-in modules between CPython, Jython and IronPython

In the last couple of years or so Python has moved to the point where it has several strong and widely used implementations. The major ones, in terms of usage, are CPython, IronPython and Jython.

(A full list of Python implementations is difficult because it really depends by what you mean by "Python". PyPy definitely counts of course, but is Stackless a separate implementation and what about PyMite, Shedskin, Cython, cl-python, Boo, CrossTwine, HotPy and many other projects that are partial or partially complete implementations of Python?)

This changes the development of core-Python (the C implementation known as CPython) slightly. As well as being the most widely used implementation CPython also serves as the reference implementation for the others.

Brett Cannon is one of the more prolific of the rag-tag bunch of developers comprising the core development team. One of his tasks has been to fully implement the Python import machinery in pure Python: importlib. importlib is already part of Python 3.1, and what will become Python 2.7, providing an import_module function.

The intention is for importlib to become the default implementation of import for Python. By being written in Python this should make it easier for alternative implementations to use - so long as they have the requisite dependencies. In this blog entries Brett looks at which of the dependencies are already available in IronPython and Jython.
But using importlib as import imposes a bootstrapping problem. How do you import, well, import? First off, you need to find the source code, compile it into a code object, and create a module object using that code object. That part is actually easy as you can simply look for the file on sys.path since you know what you are looking for, you can compile the source using the built-in compile() function, and then you finally create a module and initialize it with exec(). This is essentially what importlib does at a rudimentary level.

But import obviously goes beyond the rudimentary. There is bytecode to read and write, packages to deal with, warnings to raise, etc. And all of that requires code from some module in the standard library. But if you are trying to bootstrap in import w/o having a full-featured import, what do you do? You rely on built-in modules is what you do.

By using built-in modules you could have the VM inject any built-in module into the created importlib module and have it begin using it. Because of this I was curious as to what built-in modules CPython 3.1, Jython 2.5, and IronPython 2.6b2 had in common.

Dynamic Languages – Back to "Just Text" with Silverlight

Silverlight is the Microsoft cross platform browser plugin that lets you execute Python code (or Ruby) in the browser. When the dynamic languages integration was first announced one of the big advantages that was touted was that it was all "just text".

Like normal Python development your source files were just text files that you could edit, refresh the browser and then see the changes. When the final version of Silverlight 2 finally came out this had changed slightly. Naturally the source files are still just text but they have to be packaged in a zip file called a "xap" file. The xap contains the Python sources and assemblies and the Silverlight IronPython integration automatically executes the application once Silverlight has loaded the xap.

The development process is still straightforward as a development server called Chiron will automatically package the xap file on the fly. You run this locally, and it runs fine under Mono on the Mac or on Linux if you are using Moonlight 2, and can still edit source files and refresh the browser to immediately see your changes. You do have to package the xap file for deployment though (typically still using Chiron).

Jimmy Schementi, who is the program manager in charge of the Silverlight integration with dynamic languages, wants to bring back back "just text". Partly as a result of the gestalt project, which uses Silverlight and the Dynamic Language Runtime to allow you to use Python or Ruby files for browser scripting in the same way that you use Javascript, the IronPython 2.6 and Silverlight 3 integration has some very interesting new features. The old deployment technique will still work for backwards compatibility.

Jimmy has written up a paper describing the changes:
This document will detail the changes to the existing Microsoft.Scripting.Silverlight.dll to remove the need for Chiron.exe to simply achieve an acceptable development mode, make the development model more familiar to browser developers and bring out the “just-text” benefits of dynamic languages.

Making the default close to how browser-JavaScript development works makes Ruby or Python that much more familiar in the browser. This will also remove any obstacles to developing Silverlight applications on any operating system. For example, current Ruby or Python developers who do not use Windows have a more difficult time developing Silverlight applications, since they have to install Mono to run Chiron.

These changes will introduce no breaking changes to existing DLR-based Silverlight applications; using Chiron.exe for development and starting an application with an app.* file or the “start” initParam will still work. These changes provide a simpler default while preserving the original functionality.
One thing this restores is the ability to import external modules direct from the server rather than having to pre-package all dependencies into the xap file. This will allow for faster startup times as modules can be imported in the background or as needed - a welcome improvement.

The paper discusses the possibility of a lightweight debugger that could run in the browser and also discusses how inline Python scripts will work, interacting with both the DOM and the Silverlight control(s).

Saturday, September 12, 2009

A Good Mix 21: 3D Modelling, WMI, and IronPython Startup

Another collection of blog entries and articles related to IronPython and the Dynamic Language Runtime.
IronPython was integrated for some time to NaroCAD. But in fact it has two problems:
  • It was not able to access NaroCAD shapes or anything from NaroCAD so it was not powerful enough to do something useful, excluding you did want to make a processing task using python
  • IronPython implementation (and python in itself) do not work with Generics. So even you had access to Naro's document model, the Naro's extensible model was not accessible to IronPython script programmers.
This is why it was used lazy loading of IronPython as it does useless slowdown on startup of NaroCAD.

Right now both are addressed, exposing to you a document reference that you can create from it using a PyNodeUtil class a shape node. This is fairly important achievement as it can make easier without recompile NaroCAD to migrate your shapes if you can expose to IronPython a non generic class with your shapes.
A very brief article on .NET dynamic languages (IronPython and IronRuby) from the University of Texas at San Antonio Institute for Cyber Security:
Earlier this week, version 0.9 of IronRuby was released, the last beta before the final 1.0 version. About two weeks before that, the IronPython project released its feature complete 2.6 beta of the most widely used version of Python. The "iron" prefix implies that these languages are meant to execute on the .NET runtime and interoperate with the .NET framework and classes.
A short snippet of code using Windows Management Instrumentation (WMI) to access the system time of a remote computer.
Another newcomer to IronPython. In this blog entry Farid Zakaria explains some of the fun he has been having and then goes on to look at a few different IDE / editors for working with IronPython.
Working my way very slowly through IronPython in Action, and so far it is pretty neat. Probably the craziest thing is building a GUI application using the interactive console. However in order to get the most of out that joy, the interactive console must be placed on a separate thread. Normally it is on the same thread as the GUI thread causing it to lock up since the Form is in a continous loop.

The IronPython distribution provides a sample script winforms.py. Winforms specifically places the console on a separate thread to play around with the GUI interactively. With winforms imported, you can display the form by calling form.Show () rather than Application.Run (form).

Making changes to the GUI during runtime makes me smile.
These aren't so much blog entries as they are interesting titles... In the first Mikhael Plavskiy complains that just because IronPython does a lot on startup (imports site.py) that's no excuse for it taking a long time. I suggested that he tries with an empty site.py, or as he points out the -S command line switch, and suddenly IronPython starts up four times faster! IronPython startup improves in IronPython 2.6, but skipping the importing site step still has a big benefit.

Thursday, September 10, 2009

The Further Adventures of IronPython in Turkey

Ibrahim Kivanc is a Turkish blogger and programmer who has been having fun with IronPython and Silverlight. We've covered some of his exploits previously.

He's created a new website to promote IronPython in Turkey, but it won't make much sense unless you can read Turkish.
Fortunately for those of us who can't read Turkish, Ibrahim is intending to translate his articles into English. He says:
I'm in a progress with IronPython for a year. Last year I made presantitons with Microsoft Student Partner program and INETA program in Turkey about Ipy.In my blog I wrote more than 20 articles about IronPython in Turkish, most of them about IronPython with Silverlight working together in my blog . As soon as possible I will also translate my articles to English for foreign people.

I translated AgDLR and Chiron in Turkish. Now many Turkish people easily develop Silverlight app with IronPython. This year I will make more presentation and promote IronPython in many Turkish Colleges.
Since I last reported on him he has written two more articles on IronPython, both with example code to download:
An illustration of the Fibonacci series using IronPython, XAML and Silverlight.
The Python conditional syntax - using if, else and elif. Again illustrated with Silverlight.

Four IronPython Podcasts

It's been a while since I've posted (at least by the standards of this blog) and as usual a lot has happened. This includes four podcasts featuring IronPython.
Sod This is the product of the fevered imaginings of Oliver Sturm and Gary Short, two well known English .NET developers. In this episode they interview Harry Pierson, the IronPython program manager for Microsoft.
Wow, we managed to record another show right away this time! We talk about a variety of things – my experiences with my Snow Leopard install, 32 vs 64 bit, dynamic languages and how the world is moving on so fast… lots of talk, and during mixing it occurred to me that I was a bit too talkative myself, which makes things a bit unbalanced :-) Well, I guess that’s how it is if there’s “only” two of you.

In this episode we also include an interview we recorded with Harry Pierson at TechEd US. Harry is an IronPython guy these days, working for Microsoft as the PM on IronPython, and he had a lot of interesting stories to tell.
Python 411 is a podcast by the incorrigible Ron Stephens. In this episode he interviews Sven Passig, a programmer with comsulting.de, about an application he has been developing with IronPython and Silverlight (the Microsoft cross-platform browser plugin that lets you run Python in the browser). Interestingly Sven develops the application on a Mac using Mono.
This episode of Python411 is an interview with Sven Passig about his creation of a professional B2B app using IronPython and Silverlight on Mono and the Mac.
An interview with me with Randal Schwartz and Leo Laporte on FLOSS Weekly. The podcast is a rambling discussion over about 45 minutes. We discuss topics like, the differences between IronPython and CPython, Ironclad for using Python C extensions from IronPython, Silverlight, the Dynamic Language Runtime (DLR) and so on.
The Linux Outlaws recorded a whole bunch of interviews at this year's EuroPython conference. One of the interviews was with me, again about IronPython and Ironclad.

My part in the show is near the end and lasts about ten minutes or so but manages to cover some of the same ground as the FLOSS Weekly interview, with perhaps a bit more about IronPython on Mono.
  • 0:05:39 Steve Holden, Chairman of the Python Software Foundation (PSF) talks about the Foundation and what else he does in the open source world
  • 0:13:50 Laura Creighton from Open End talks about the EuroPython conference and also PyPy
  • 0:36:12 Bruce Eckel talks about writing technical books, writing them collaboratively and making books available for free on the web
  • 0:55:18 Michael Foord from Resolver Systems talks about IronPython and what he does with it at his work (including Ironclad) - also check out his book IronPython in Action