Distributing IronPython Packages: distutils, site-packages and PEP 370
The basic way of distributing and installing Python libraries (packages), and even scripts and applications, is with distutils which is part of the Python standard library.
A Python package consists of various source files and a setup.py script, which when executed at the command line installs the Python package into your site-packages folder. The typical invocation of a setup script is:
Unfortunately distutils doesn't work with IronPython, for at least two reasons.
The second reason, the most easily dealt with, is that attempting to install a package with IronPython causes distutils to attempt to bytecode-compile the Python files; something destined to fail with IronPython.
The first problem is that a normal installation attempt will fail like this:
CPython on Windows makes its site-packages folder writable by non-privileged users. This allows distutils to work its magic, but is also the cause of some of the odd behavior of easy_install; where it has to run a second cmd window with elevated privileges to actually install and this window flashes before your eyes not giving you time to read any error messages displayed. The solution is to elevate before running easy_install.
Even worse the CPython behaviour could be seen as a security issue. An untrusted user could create a sitecustomize.py that checks if the user is admin and installs malware. All the admin has to do is run Python and the code is executed with her priveleges.
It seems like the behavior of IronPython is correct and that CPython should change. Unfortunately that would mean the same access denied errors when installing packages unless done from an elevated command line (elevation of individual commands can also be done from the command line in a similar way to sudo but using runas).
There is fortunately a solution, both for CPython and for IronPython, in the form of PEP 370 per user site-packages folders that are new in Python 2.6.
If you install CPython 2.6 on Windows it creates a folder in the location: %APPDATA%\Python\Python26\site-packages. On my machine the full path is: C:\Users\michael\AppData\Roaming\Python\Python26\site-packages.
This is put on the path by site.py when you run Python and as it is a user directory it doesn't require elevation to install there. You install into the user directory rather than the 'usual' location with:
The best solution would be for site.py in Python 2.6 to have a compatibility fix applied where it conditionally adds an implementation specific folder to the path instead. distutils would also need to be aware of this so that when the --user flag is used packages are installed into the correct location. It would be a nice touch if --user could be the default in IronPython, but this isn't essential.
In Python 2.7 the sys module could grow a new attribute, perhaps sys.usersitepackagespath, so that the logic to set the right location can be moved out of site.py and distutils and back into the core implementations. (Antoine Pitrou has kindly volunteered to do this if it is acceptable to the other core-devs.)
UPDATE: Christian Heimes, the author of PEP 370, is suggesting that sys.vm (or similar) should hold the name of the implementation and be used by site.py and distutils to determine the location of the user site-packages folder.
As a further step, the holy grail of managing IronPython packages will be getting Distribute (the successor to setuptools which is now essentially unmaintained) to run on IronPython. The maintainer of both Distribute and distutils is Tarek Ziade who is not only doing an awesome job but also very open to working on IronPython compatibility if he has help. One stumbling block is that Distribute (like its predecessor) is not yet PEP-370-aware, but this will be fixed in due course. Other stumbling blocks are the lack of zipfile and subprocess standard library modules in IronPython. Both of these require C extension modules not available for IronPython. The IronPython team will (at some point) get round to these, but in the meantime there are open source alternative implementations that Distribute could ship for use only on IronPython.
I'm sure there are plenty more issues, both big and small, but none are insurmountable. The goal, easily achievable, is to have quality package management tools that can be used across all the major implementations of Python.
I have filed a Python issue for the distutils problem and will work with Tarek and the other Python devs on the other compatibility issues I've discussed here.
A Python package consists of various source files and a setup.py script, which when executed at the command line installs the Python package into your site-packages folder. The typical invocation of a setup script is:
python setup.py installsite-packages is a directory created as part of a standard Python installation specifically to hold user installed Python packages (glossing over the difference between user and system installed packages for the purposes of this discussion).
Unfortunately distutils doesn't work with IronPython, for at least two reasons.
The second reason, the most easily dealt with, is that attempting to install a package with IronPython causes distutils to attempt to bytecode-compile the Python files; something destined to fail with IronPython.
This is a bug in distutils that is easily remedied; under IronPython sys.dont_write_bytecode is True.C:\compile\test\foo-1.0>ipy.exe setup.py install running install running build running build_py running install_lib copying build\lib\foo.py -> C:\Program Files\IronPython 2.6\Lib\site-packages byte-compiling C:\Program Files\IronPython 2.6\Lib\site-packages\foo.py to foo.pyc Traceback (most recent call last): ... ValueError: unmarshallable object
The first problem is that a normal installation attempt will fail like this:
C:\compile\test\foo-1.0>ipy.exe setup.py installThis is a permissions issue. The default install location for IronPython is "C:\Program Files\IronPython 2.6". Installing to this location requires elevated permissions and the site-packages folder it creates is not writable by non-admin users. If you run the same command from an elevated prompt (running with elevated permissions is the Windows equivalent of sudo) then it gets as far as failing with problem 2. The error message prompted by the "access denied" failure could certainly stand to be improved though.
...
error: ...foo-1.0-py2.6.egg-info: None
CPython on Windows makes its site-packages folder writable by non-privileged users. This allows distutils to work its magic, but is also the cause of some of the odd behavior of easy_install; where it has to run a second cmd window with elevated privileges to actually install and this window flashes before your eyes not giving you time to read any error messages displayed. The solution is to elevate before running easy_install.
Even worse the CPython behaviour could be seen as a security issue. An untrusted user could create a sitecustomize.py that checks if the user is admin and installs malware. All the admin has to do is run Python and the code is executed with her priveleges.
It seems like the behavior of IronPython is correct and that CPython should change. Unfortunately that would mean the same access denied errors when installing packages unless done from an elevated command line (elevation of individual commands can also be done from the command line in a similar way to sudo but using runas).
There is fortunately a solution, both for CPython and for IronPython, in the form of PEP 370 per user site-packages folders that are new in Python 2.6.
If you install CPython 2.6 on Windows it creates a folder in the location: %APPDATA%\Python\Python26\site-packages. On my machine the full path is: C:\Users\michael\AppData\Roaming\Python\Python26\site-packages.
This is put on the path by site.py when you run Python and as it is a user directory it doesn't require elevation to install there. You install into the user directory rather than the 'usual' location with:
python setup.py --user installHerein lies another problem though. IronPython uses the standard Python site.py, meaning that currently it shares the user site-packages folder with CPython. This is not ideal as typically you will want separate libraries installed for IronPython. When Jython ports to Python 2.6 it will have the same issue.
The best solution would be for site.py in Python 2.6 to have a compatibility fix applied where it conditionally adds an implementation specific folder to the path instead. distutils would also need to be aware of this so that when the --user flag is used packages are installed into the correct location. It would be a nice touch if --user could be the default in IronPython, but this isn't essential.
In Python 2.7 the sys module could grow a new attribute, perhaps sys.usersitepackagespath, so that the logic to set the right location can be moved out of site.py and distutils and back into the core implementations. (Antoine Pitrou has kindly volunteered to do this if it is acceptable to the other core-devs.)
UPDATE: Christian Heimes, the author of PEP 370, is suggesting that sys.vm (or similar) should hold the name of the implementation and be used by site.py and distutils to determine the location of the user site-packages folder.
As a further step, the holy grail of managing IronPython packages will be getting Distribute (the successor to setuptools which is now essentially unmaintained) to run on IronPython. The maintainer of both Distribute and distutils is Tarek Ziade who is not only doing an awesome job but also very open to working on IronPython compatibility if he has help. One stumbling block is that Distribute (like its predecessor) is not yet PEP-370-aware, but this will be fixed in due course. Other stumbling blocks are the lack of zipfile and subprocess standard library modules in IronPython. Both of these require C extension modules not available for IronPython. The IronPython team will (at some point) get round to these, but in the meantime there are open source alternative implementations that Distribute could ship for use only on IronPython.
I'm sure there are plenty more issues, both big and small, but none are insurmountable. The goal, easily achievable, is to have quality package management tools that can be used across all the major implementations of Python.
I have filed a Python issue for the distutils problem and will work with Tarek and the other Python devs on the other compatibility issues I've discussed here.
Hi Michael!
ReplyDeleteAs you most likely know I'm the author of PEP 370. I didn't thought about IronPython when I designed the feature. I've a nice and simple idea how to solve the issue. I'm going to post the idea later on the Python developer list. Stay tuned!
Christian