Saturday, November 29, 2008
There is good news for IronPython 2. Binary compilation is much more efficient as we can compile multiple packages into a single binary and then ngen (pre-JIT), making imports faster.
Whilst exploring how to improve our startup time Kamil Dworakowski experimented with a system for performing parallel imports on multiple threads.
import lock as CPython, so you need to ensure that parallel imports don't pull in the same modules simultaneously or some of the imports may fail. To get round this Kamil created a modified modulefinder that analyses a codebase and generates a dependency graph for all the imports (done at 'compile time' not runtime). So long as you start at the leaves of the graph you know you are safe. You can then configure how many threads the parallel importer should use in its threadpool to do the imports.
On a multi-core machine this yields substantial improvements for import time. Unfortuately there are a few thread safe issues for importing in IronPython 1, leading to intermittent crashes when starting Resolver One. We weren't able to use Kamil's code for Resolver One 1.3, but the specific problems we encountered are fixed in IronPython 2 so we might be able to use it in our next release...
Even though we haven't yet been able to take advantage of Kamil's work, others have. Dan Eloff, who is building a Silverlight gaming platform with IronPython and C#, has used it. He had this to say on the IronPython mailing list:
Thanks to the excellent work of Kamil and the help of Jimmy I now have importing being done in parallel in 4 threads in Silverlight.
I had to restructure my code a little to reduce circular imports (not
a bad thing to do anyway) and I had to hand-tweak the dependency graph to deal with conditional imports and other difficult to track
The result was better than I could have hoped for. I'm seeing 43%
faster loading on my dual-core development pc. I used to have blank screen at start that made you wonder if the browser has frozen. But now the browser is responsive and I have a nice working progress bar. But that's not the best news.
The kicker is that the benefit is even larger on slow single core
machines with dialup internet. I posted a while back on this list that
IE does not immediately send an asynchronous request. It waits until its UI thread is idle (which doesn't happen if your importing modules like crazy on it.) So the net effect was the application loads (about 28 seconds on a P4 2ghz) and then it sends the request, waits for the response (which is big in my case), and processes it - all in order.
So if that takes another 30 seconds, you have nearly 1 minute of load time. With the UI thread mostly idle now, this happens in parallel to the importing, and the overall load time drops by a whopping 45%.
In the words of my generation - w00t!
Friday, November 28, 2008
NWSGI is a .NET implementation of the Python WSGI specification using IronPython. It is implemented as an ASP.NET HttpHandler. The main goal is to provide an easy path for running Python web applications/frameworks (such as Django, Trac, etc.) on IIS7.
He recently released a new version of NWSGI:
New features in this release:
- The major new feature is IIS7 management integration, including a UI editor. It makes configuring NWSGI a breeze from the command line (appcmd) or from the UI. The IIS7 integration is only available through the installer.
- There is one minor feature addition: wsgi.file_wrapper is now supported for faster file transfers. It uses HttpResponse.TransmitFile to send the file directly.
The changes are in the configuration system to support the IIS7 management system. Unfortunately, IIS7 doesn't use System.Configuration; instead, it has its own assembly, Microsoft.Web.Administration. This caused some complications for configuration. There are now three possible usage scenarios (IIS6, IIS7 xcopy, and IIS7 installed) that require slightly different web.config settings; I will detail these in an upcoming post.As for using Django with IronPython and NWSGI, Jeff has some interesting ideas (and demo code to go with it) on ASP.NET features that might actually be useful for Django:
A demo version of django-aspnet is now available...
Azure is Microsoft's new cloud computing platform. If you're interested in using IronPython and NWSGI on Azure, Jeff has been looking at that too:
NWSGI 0.6 actually supports medium trust. I only discovered after the release that the error I was hitting was a bug in Cassini; it works fine in medium trust on IIS7. With that hurdle out of the way, there's no reason NWSGI shouldn't work on Azure.
And, what do you know, it works just fine: NWSGI "Hello, World" for Windows Azure. You'll need the Azure SDK and the Visual Studio tools to run it.
Latest sources migrated to CodePlex TFS", rather than actually telling you anything useful about what has changed.
He also explains how their processes have changed, enabling them to do code syncs to the public repository more often. (Something that is very useful for us at Resolver Systems as we port Resolver One to IronPython 2. We are using the new source drops daily so that we can pick up problems as quickly as possible.)
Thursday, November 27, 2008
The latest release is version 0.7 and includes several major advances since the last release.
- Huge chunks of numpy can now be imported and used
- Lots of the numpy tests now pass (from core, fft, lib, linalg and random subpackages) - the distribution includes a test runner that runs the tests we have enabled so far
- Major performance improvements (several orders of magnitude faster in some areas)
- Ironclad works with IronPython 2 RC 2 and it no longer creates a separate Python engine, nor does it break the importing of .NET libraries!
C:\compile\ironclad-v0.7-bin\build>"C:\Program Files\IronPython 2.0\ipy.exe"
IronPython 2.0 (18.104.22.168) on .NET 2.0.50727.1433
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import ironclad
>>> import numpy
Detected numpy import
faking out modules: nosetester, parser, mmap, urllib2, ctypes
>>> a = numpy.array([[1, 2, 3], [4, 5, 6]])
>>> a[1, 1]
>>> a = numpy.arange(60).reshape(3, 4, 5)
array([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]],
[[20, 21, 22, 23, 24],
[25, 26, 27, 28, 29],
[30, 31, 32, 33, 34],
[35, 36, 37, 38, 39]],
[[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49],
[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59]]])
>>> a[..., 3]
array([[ 3, 8, 13, 18],
[23, 28, 33, 38],
[43, 48, 53, 58]])
So much progress has been made in this release that Giles Thomas hints that "we’re now seriously considering having it as an option (with an explicit note that it’s not ready for production use) in the next release of Resolver One".
Bugs fixed in this release include:
- 19350: Missing restrictions on index operation against user defined type
- 11733: time.timezone has different semantic than in cpython
- 19675: subclasses of float have trouble with __int__ and __str__
- 19656: Module Name Lookup Broken
- 19678: in operator calls __getitem__ on class that has __len__ and __iter__ defined
- 19665: operator.isSequenceType(NewStyleClass) broken
- 19130: One hour error in IPY implementation of time.mktime and time.gmtime
- implement a language on .NET using the DLR
- add dynamic features to their existing language like C#’s ‘dynamic’
- add scripting to their applications.
- create .NET libraries with dynamic objects
- fast dynamic dispatch with polymorphic inline caching
- dynamic object interoperability across languages and libraries (including C# ‘dynamic’ consuming IronPython and IronRuby objects naturally, including any language whose objects participate in the DLR’s dynamic object protocol)
- support for library authors to easily make their model objects consumable with nice looking and lightweight code in languages that support the DLR dynamic object protocol (for example, with C#’s ‘dynamic’, you can write xml.Customer.Name instead of xml.GetChild(“Customer”).GetChild(“Name”)
- ability to mix binding logic from various languages and library objects in a single dynamic call site cache
- Expression Trees v2 with support for control flow, assignments, etc.
- COM IDispatch binding as DLR dynamic objects
- common hosting model for languages built on or supporting the DLR hosting model
- helpers such as a default .NET binder, complex numbers, tuples
- more Expression Tree support, such as globals access in hosted scenarios and iterator/generator functions.
IronPython Program Manager is excited about the release, and describes it as Early Christmas from Iron Languages and DLR :
Tomorrow may be Thanksgiving, but the Microsoft DevDiv dynamic language teams are trying to make it feel like Christmas with three separate pre-holiday releases.
So there you go, new versions of IronPython and IronRuby plus a whole new DLR CodePlex project to boot. Enjoy.He notes that along with the IronPython and DLR releases, there has also been a new release of IronRuby:
IronRuby 1.0 Alpha 2
There’s been zero blog traffic on this, just a notice on the IronRuby mailing list. As per said notice, “Notable features” include “the inclusion of iirb.bat, igem.bat, irails.bat, irake.bat”.
Seshadri (one of the DLR testers) promises regular binaries and weekly source drops in his blog entry on the DLR release.
We do not test the list box component. We test the application.
When testing an application you don't need to test the functionality provided by the operating system or the frameworks you're using; you test your application logic.
Example code for selecting an item in a list box from the application under test (is SUT for "System Under Test" a commonly used acronym?):
def Select(self, aItem):
""" select an item from list box
@aItem - string with item to select
if aItem not in self.items:
raise Exception("Item '%s' not in list box" % aItem)
pos = self.location
# click on the first item to focus the list box
Win32API.MouseClick(pos + (pos/2), pos + 3)
# send Home to be sure we are on the first item
# (we could be scrolled down a little)
# simulate pressing down arrow until we find the item
# we should find it because it is among self.items
while self.value != aItem:
Friday, November 21, 2008
from System.Data.SqlClient import SqlConnection, SqlParameter
conn_string = 'data source=
; initial catalog= ; trusted_connection=True'
connection = SqlConnection(conn_string)
command = connection.CreateCommand()
command.CommandText = 'select id, name from people where group_id = @group_id'
reader = command.ExecuteReader()
print reader['id'], reader['name']
Thursday, November 20, 2008
Oleg Tkachenko lists them on his blog:
- John Lam – IronRuby PM
- Harry Pierson – IronPython PM
- Jimmy Schementi – PM for Silverlight integration
- Shri Borde – dev lead for the team
- Dave Remy – PM lead fro the team
- Oleg Tkachenko – Visual Studio integration
- Dave Fugate – IronPython tester
- Curt Hagenlocher - IronPython/IronRuby dev
- Jim Hugunin – architect for the team, DLR and Visual Studio languages
- Tomas Matousek – IronRuby dev
- Srivatsn Narayanan – IronPython/IronRuby tester
- Jim Deville – IronRuby tester
- Dino Viehland – IronPython dev
Wednesday, November 19, 2008
Silverlight 2 final is now out, and according to Scott Guthrie has now been installed on over 100 million consumer computers.
Naturally Jimmy Schementi, who maintains the Dynamic Language Support for Silverlight, released an updated version of the Silverlight Dynamic Languages SDK (sucky name - more on this in a bit):
PyWorks conference I updated my IronPython Web IDE (tool for experimenting with the Silverlight APIs from Python in the browser) and Try Python (interactive Python interpreter in the browser) for Silverlight 2:
Visual Studio Tools for Silverlight 2 (this works with Visual Studio Professional or the free Visual Web Developer Express). The assemblies that come with the tools include the data grid, extra controls like the date picker and various other useful APIs.
Even better is the Silverlight Toolkit. This is by Microsoft, but Open Source (living on Codeplex and with Unit Tests). As it is a separate project it can have a separate release cycle, including experimental components and being updated more frequently than Silverlight itself.
The toolkit includes charting components plus new controls covering styling, layout, and user input.
Since these releases Jimmy Schementi has been far from idle. His latest blog entries track what he has been up to:
First and foremost, I want to thank anyone who has used the bits on http://codeplex.com/sdlsdk, and accepting my bullshit version of open-source. While getting monthly binaries/sources is nice, it should be about working on the project together ... not just me throwing stuff over the wall to you. That's changing, now ...
Oh, and remember me complaining about the crappy "sdlsdk" name ... well, I'm trying to get rid of that acronym ...
Above the public repository for the DLR integration in Silverlight. The following post explains what's in there, what's not, what's git, and how to contribute: http://blog.jimmy.schementi.com/2008/11/agdlr-silverlight-dlr-open-source.html
My first order of collaboration is this simple new feature, "console=true": http://blog.jimmy.schementi.com/2008/11/repls-in-silverlight.html. If you like this, please feel free to look at what's been done, and if you want to fix something that doesn't yet work correctly, I won't stop you.
Also, as I mentioned in a previous mail, I want to make the filesystem->XAP/isolatedstorage metaphor stronger, so feel free to experiment with that as well. Over the next week I'll get some website-presence/wiki/etc, and we can run this project up and running. There are still some hurdles I need to clear with getting contributed code back into our internal codebase, and shipping on Codeplex, but there are no problems with keeping things on GitHub for now.
Let me know if there are any question. I know I've been a bit silent on the Silverlight front, but take this as me making it up to you.
The resulting discussion also revealed where the Silverlight development tool Chiron got its name from:
Yep, Ag is Silver … made pretty obvious by my little logo for it.
As far as Chiron. It's a planetoid between Saturn and Uranus. The port that Chiron.exe runs on by default, 2060, is Chiron's "object" number. It was derived from the Cassini ASP.NET Web server that Dmitry Robsman wrote. Cassini was a probe mission to explore the moons of Saturn, and Chiron was initially thought to be a moon of Saturn.
Plus this from Michael Letterle:
More importantly, it's also the name of one of Jonathan Coulton's songs: "Chiron Beta Prime".
Because of this I knew how to pronounce the name :)
Tuesday, November 11, 2008
IVisio = Microsoft.Office.Interop.Visio
visapp = IVisio.ApplicationClass()
doc = visapp.Documents.Add("")
page = visapp.ActivePage
shape = page.DrawRectangle(1, 1, 5, 4)
shape.Text = "Hello World"
After only a few days of playing with Python, I see now what others have been raving about. Web developers will really enjoy using Python with Dialogue Script:
- Cleaner code. Easier to read and manage
- No need for thick IDEs. Too often, an IDE like Eclipse or Visual Studio stands between you and the desired solution. Dialogue Script development is 100% browser-based (yes, it even works in Chrome!)
- Agile business rules management. Work side-by-side with business users and apply business rules directly in web pages
- Dynamic typing
Monday, November 10, 2008
The gist of this presentation is that it's possible to mix the Dynamic Language Runtime (DLR) into statically-typed, early-bound languages like C# to make them much more flexible. In this talk, I demonstrated how a ShoppingCart being filled with Products can adjust discount rates based on marketing rules written in an external Domain Specific Language (DSL). In this case, my DSL was really just Python. I chose to use Python because the syntax is so simple and clean. It's so light, it doesn't get in the way. It's not a real DSL, of course, but by injecting .NET objects into a ScriptScope on a ScriptRuntime (all DLR hosting terms), the Python syntax acting on those injected types looks an awful lot like a language for managing product discounts."
Slides and code examples are linked to from the blog entry.
UPDATE: Kevin has done a 'November Update' to this presentation, for an upcoming Raleigh Code Camp talk. The new blog entry has screenshots and updated code to download:
"Python code is pretty easy to read isn't it."
"One of the most often asked questions I get when talking about IronPython is, "How do I convince my boss to let me write that application in Python?" My answer is, "You don't have to write the whole thing in Python. With the fantastically rich hosting APIs in the .NET Dynamic Language Runtime (DLR), you can write the portions that need dynamism in Python and write the rest in C# or VB.NET."
winforms sample that comes with IronPython:
from System.Windows.Forms import *
f = Form()
f.Text = "GeckoFX"
from Skybound.Gecko import *
browser = GeckoWebBrowser()
browser.Parent = f
browser.Dock = DockStyle.Fill
codeplex example (except without cookies...but the addition should be trivial). It uses the WebRequest class.
Points to note are: 1) the parameter query_string is of type NameValueCollection, 2) the parameter url must be a valid URI (e.g. www.somewebsite.com/somepage), 3) the parameter content_type can be 'multipart/form-data'.
IKVM, an implementation of Java for .NET.
The command line magic to compile CDK for .NET is:
ikvmc -assembly:cdk_dotnet -target:library yourcdkjar.jar
This produces a .NET assembly. An example of using the library from IronPython:
from org.openscience.cdk import Molecule
from org.openscience.cdk.smiles import SmilesParser
#import the whole package for brevity
from org.openscience.cdk.qsar.descriptors.molecular import *
from org.openscience.cdk.qsar.result import DoubleResult
tpsa = TPSADescriptor()
logP = XLogPDescriptor()
smiles = SmilesParser()
mol = smiles.parseSmiles("N=CCC=O")
dr = tpsa.calculate(mol).getValue()
tpsaVal = dr.doubleValue()
dr = logP.calculate(mol).getValue()
logPVal = dr.doubleValue()
the output is:
Sunday, November 09, 2008
- All of the .NET libraries are available.
- UIs look nice. I’ve never seen a pure traditional Python application that looked good, no matter how advanced its functionality.
- We use a bunch of third-party components - for example, Syncfusion’s Essential Grid - without any problems.
- Reasonably decent multithreading using the .NET libraries - CPython, the normal Python implementation, has the problem of the Global Interpreter Lock, an implementation choice that makes multithreading dodgy at best.
- We can build our GUI in Visual Studio, and then generate C# classes for each dialog, and then subclass them from IronPython to add behaviour. (We never need to look at the generated code.)
- When things go wrong, the CLR debugger works well enough - it’s not perfect, but we’ve never lost a significant amount of time for want of anything better.
Another interesting post is his description of how we're transitioning development at Resolver Systems away from being a single team into two. This is due both to growth in the development team and the need for us to do development work for some of our larger customers. The entry includes an example Resolver One spreadsheet that uses an evolutionary algorithm to do scheduling to spread developers across the two teams over time:
IronPython in Action.
The techniques and testing patterns used in the articles are not tied to IronPython at all. The topics and principles discussed are relevant to the testing of all desktop applications, whichever framework you are using (almost...). Topics covered include:
- The why and how of functional testing - including the processes and infrastructure you need around them
- Basic principles of practical testing
- Common problems and ways to overcome them