When having trouble while installing an egg during the installation phase(easy_install time)

Now the recipe can display some information about which wanted this distribution.

This is even more useful that it support up to 6 dependencies levels from direct to parents !

Here is an example:

Processing ssl-for-setuptools-1.10.tar.gz
Unpacking ssl-for-setuptools-1.10 to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10
Unpacking ssl-for-setuptools-1.10/PKG-INFO to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/PKG-INFO
Unpacking ssl-for-setuptools-1.10/setup.py to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/setup.py
Unpacking ssl-for-setuptools-1.10/ssl to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/ssl
Unpacking ssl-for-setuptools-1.10/ssl/2.3.6 to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/ssl/2.3.6
Unpacking ssl-for-setuptools-1.10/ssl/2.3.6/socketmodule.h to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/ssl/2.3.6/socketmodule.h
Unpacking ssl-for-setuptools-1.10/ssl/2.5.1 to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/ssl/2.5.1
Unpacking ssl-for-setuptools-1.10/ssl/2.5.1/socketmodule.h to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/ssl/2.5.1/socketmodule.h
Unpacking ssl-for-setuptools-1.10/ssl/__init__.py to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/ssl/__init__.py
Unpacking ssl-for-setuptools-1.10/ssl/_ssl2.c to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/ssl/_ssl2.c
Unpacking ssl-for-setuptools-1.10/test to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/test
Unpacking ssl-for-setuptools-1.10/test/badcert.pem to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/test/badcert.pem
Unpacking ssl-for-setuptools-1.10/test/badkey.pem to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/test/badkey.pem
Unpacking ssl-for-setuptools-1.10/test/https_svn_python_org_root.pem to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/test/https_svn_python_org_root.pem
Unpacking ssl-for-setuptools-1.10/test/keycert.pem to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/test/keycert.pem
Unpacking ssl-for-setuptools-1.10/test/nullcert.pem to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/test/nullcert.pem
Unpacking ssl-for-setuptools-1.10/test/test_ssl.py to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/test/test_ssl.py
Running ssl-for-setuptools-1.10/setup.py bdist_egg --dist-dir /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/egg-dist-tmp-NfSQin
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 1714, in main
    with_ei_usage(lambda:
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 1695, in with_ei_usage
    return f()
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 1718, in <lambda>
    distclass=DistributionWithoutHelpCommands, **kw
  File "/usr/lib/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 236, in run
    self.easy_install(spec, not self.no_deps)
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 452, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 501, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 680, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 957, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 946, in run_setup
    run_setup(setup_script, args)
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/sandbox.py", line 29, in run_setup
    lambda: execfile(
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/sandbox.py", line 70, in run
    return func()
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/sandbox.py", line 31, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 11, in <module>
    def read(rnames):
ValueError: This extension should not be used with Python 2.6 or later (already built in), and has not been tested with Python 2.3.4 or earlier.
    - Failed specs: ssl-for-setuptools==1.10
    - required by:
        - [Requirement.parse('zc.ssl==1.1')]
    - required by:
        - [Requirement.parse('zc.authorizedotnet==1.3')]
    - required by:
        - [Requirement.parse('easyshop.core==0.1a1')]
    - Failed specs: ssl-for-setuptools==1.10
    - required by:
        - [Requirement.parse('zc.ssl==1.1')]
    - required by:
        - [Requirement.parse('zc.authorizedotnet==1.3')]
    - required by:
        - [Requirement.parse('easyshop.core==0.1a1')]
While:
  Installing zopepy.

We see at first shot that easyshop give us trouble ! Without, its hard to know which distribution want ssl-for-setuptools :)

While upgrading my gentoo based laptop after 8monthes of lazy abandonness in profit of exiting projects, i saw that the gentoo's python was forced to use UCS==4.

What an heck while dealing with the 'minitage env' file which mix the system and project environment resulting in a mixin of the system and project python.

Normally, there are no problem, unless your pythons come with different UCS flavors.

Cool thing is that those errors are not silent, and you see them if you are hitted by this flaw :

    ImportError: /bar.egg/module/_foo.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8
or  
    ImportError: /bar.egg/module/_foo.so: undefined symbol: PyUnicodeUCS2_DecodeUTF8

Making some searches showed me that upstream as in python-dev supports only UCS2 by default (issue discussed on their mailing around 2008) but all distros i am aware of package their distros with UCS==4.

For now, I prefered to stick with upstream (UCS==2), but i think that for the user experience, UCS==4 will be better.

Sad thing is that for existing minitage installations if you want to rebuild your python, do also that:

rm -rf minitage/eggs/cache/*-Major.minor*egg
rm -rf minitage/eggs/*/.installed.cfg

To let minitage rebuild any stuff using UCS2.

See http://en.wikipedia.org/wiki/Universal_Character_Set for reference on UCS.

I refreshed a little the ohloh minitage's related page to have some marketing analysis around minitage basecode.

For this, i wanted to add all repositories, what a heck for 200 repos...

Time to create minitage.ohloh to create the repositories for me using some fake browser ;-).

This package uses lxml (XPath), zope.testbrowser (fake browser) and urllib (some handmade http requests) to make the work done.

with more than 100 repos created :D !

 And sorry ohloh Team for flooding ohloh.net :p

.

Calling strftime on a date/datetime instance on py24/py26 will raise a value error like this:

>>> from datetime import date
>>> date(1800,1,1).strftime('%d%m%Y')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: year=1800 is before 1900; the datetime strftime() methods require year >= 1900

Quite annoying heh ?

The work here was to integrate the working patch in the bugreport to py24 and py26 python packages which are in minitage.

So now, with minitage's pythons, you can work with very old dates by default !!!

Just issue:

virtualenv --no-site-packages --distribute minitage
source minitage/bin/activate
minimerge -s
minimerge -v python-2.6 (or 2.4)

And you can use the following interpreter which support those old ages:

minitage/python-2.6/parts/part/bin/python

And of course the related patch does not break anything:

$ svn info
URL : http://svn.python.org/projects/python/branches/release24-maint
Révision : 75208
$../parts/part/bin/python Lib/test/test_datetime.py
...
----------------------------------------------------------------------
Ran 230 tests in 0.858s
OK


$ pwd
/home/kiorky/minitage/dependencies/python-2.6/Python-2.6.4
$../parts/part/bin/python Lib/test/test_datetime.py
...
----------------------------------------------------------------------
Ran 245 tests in 0.791s
OK

GIVE ME ONLY DEPENDENCIES

You can know select to build all dependencies but not the package.

For exemple, to build all python-2.4 dependencies but not python-2.4 itself, you can do:

minimerge -v --only-dependencies python-2.4

BUILD THIS PYTHON PACKAGE FOR THIS PYTHON

For 'eggs' packages, the default was to make come in dependencies all python avaiable in minitage and to build the 'site-packages-Major.Minor' part if it was a buildout build..

It is know rearranged to be a little more subtile.

Look the following:

minimerge -v mypythonpackage # will be mypythonpackage against all python as usual
minimerge -v mypythonpackage python-2.4# will be mypythonpackage against only python-2.4

Just a little post to say that snowleopard let run minitage.

As a note, you must have all SDK installed, specially the 10.4!

Now, nginx and apache are packaged inside minitage.

You have :

• two templates initiating instance of both webservers in minitage.paste.extras
• the related packages to be build via minitage (nginx-0.8, apache-2.2)

Enjoy running:

easy_install -U minitage.paste
paster create -t minitage.instances.nginx myproject #deploy some nginx instance (init scripts, logrotates, docroot)
paster create -t minitage.instances.apacher myproject #deploy some nginx instance (init scripts, logrotates, docroot)

I have added recently the support for standalone tomcat instances and running applications in those tomcat instances.

For example, to initiate a great hudson (contineous integration), you can run the following:

If you are not inside a minitage environment:

easy_install -U minitage.paste.extras
paster create -t minitage.instances.tomcat myproject # it will ask you for the location of a tomcat extracted tarball
paster create -t minitage.instances.hudson myproject

And if you are not inside a minitage environment, to deploy in a myproject subdirectory

easy_install -U minitage.paste.extras
paster create -t minitage.instances.tomcat myproject inside_minitage=no 
paster create -t minitage.instances.hudson myproject inside_minitage=no

Requirements:

• A tomcat tarball extracted somewhere (CATALINA_BASE for knowsers)
  
• A java installed somewhere with a JAVA_HOME environment variable pointing to it
  

Making more and more complicated instances from server dependencies installed into the minitage tree make me relying on somehow intrusive dependencies.

 As i want minitage.paste to depend on barely nothing else that paster and the minitage core,  i have splitted such 'instances' in an additionnal egg.

 minitage.paste.extras was born.

This package borrow 2 instances at the moment:

• minitage.instances.cas: Install a CAS server
• minitage.instances.openldap: Install an openldap server

Those instances make some nice things as generating on the flag SSL certificates for enabling secured connections to your instance(s).

There is pretty much a long time i wanted to switch my stuff on the BSD License.

More as an habit than a conviction, my source files where header pollutated by the GPL License.

For me, and in my opinion, the GPL is not as free as it claims to.

Why ? Because the final user cannot do what he want with your code.

Some debate will qualify that as communism  (GPL) vs anarchy (BSD). I prefer anarchy :).

Of course i use the "modified BSD license" which is compatible with the GPL.

Some on the concerned:

• minitage.core
• dj.paste
• minitage.recipe
• minitage.paste
• cluereleasemanager.paste
  

As part of a new project deployment plan, we needed to deploy mapscript.

Python-MapScript are the python bindings to the underlying mapserver library.

Where the pain begins is that there is no egg available for it, but an old fashioned distutils distribution;.

My bits there were to make some quickly egg compatible on most unixes. My will is to make it nicer and be integrated on the mapserver trunk code.

You can find the code there : http://git.minitage.org/git/others/mapscript/

And play with the egg:

easy_install mapscript 

You must know that mapserver-config must be available in your $PATH.

It is now an hassle to build all the minitage dependencies by hand.

Naturally, the idea to set up some tinderbox like buidbot comes to my mind.

More over, i had a though of generating some binaries to avoid the compilation time for users who don't want to/can't compile.

The lead was to use collective.buildbot, a great recipe to integrate buildbot with zc.buildout deployment.

The result is there : http://git.minitage.org/git/minitage/buildbot/

The idea was to generate a "collective.buildbot" project per dependency. To achieve that, i made a script to generate those projects from the current official minilays.

Compilation is triggered via manual activation (irc, web) but also by a cron scheduler (nearly each 3 hours)

I made a patch to the recipe to have multiple hours values (* 1,2,3 * * *) cron entries, see http://git.minitage.org/git/minitage/buildbot/tree/collective.buildbot.multiple.patch

The buildbot can be viewed there : http://www.minitage.org/buildbot/waterfall

Next steps are to finnish the "binaries dealing" in minitage.core and the mirrors setup, and minitage users will be able to use binaries and no compilation anymore :)

I made for a workmate yet another django paste's http factory.

The goals were to integrate its way to work, with setting some DJANGO_SETTINGS_MODULE key in the environment and launch the "bidule", and letz play.

The challenge was, ..., to be simple or this workmate will never use this stuff. She was used to tell only about django-admin runserver :).

One part of the challenge was to show some interessant things that we can do with paste configurations, like having different but closer configurations for production or development mode.

Thus, including showing the nice "WebError#eval" application where you can debug your messy code though the web with an interactive debugger.

But, right to the topic, the dj.paste stuff:

• It enables you to have multiple tracs in the same pipeline
• You can integrate it in an exiting paste configuration, it's the goal after all :)
• It 's simple to use, an a sample paste configuration can be:
  

[composite:main]
use = egg:Paste#urlmap
/ = foo

[app:foo]
use=egg:dj.paste
django_settings_module=foo.settings

A teamate wanted to have a binary like the package manager he was used to use.

Its done, now instead of calling minimerge, you can use either:

• minimerge
• yasty
• apty
• pacmerge
  

During last weeks, i have switched to git minitage, and by default all come from it.

The failure here is that my git fetcher is not very sympathic with older versions of git (<1.5).

But on debian and red hat legacy, it's a pain in the ass to have relativly new stuff !!!

To resolve that, with workmates, we come to the decision to fetch packages over http.

And now, it's done. Minimerge codebase is 'HttpCompliant' in sense that defaults minilays and packages are fetched via the minitage static fetcher.

This one just use urllib, and you can even set proxies in your minimerge.cfg

To be up-to-date:

easy_install -U minitage.core
mv etc/minimerge.cfg etc/minimerge.cfg.old
cd minilays
mkdir ../old
mv meta misc eggs dependencies ../old
minimerge -s

I m actually developping some application around XMLRPC protocol at work.

We are using Pylons for the framework part, and i played this afternoon at setting up some testing environnement for doing doctests.

This test is a proof of concept, it 's code extracted from our internal application, it's just a starter for you. The whole is working with some tweaks.

controllers/mycontroller.py, a simple controller doing simple stuff

class MyController(XMLRPCController):
    """controller."""                           

    def index(self):
        return '\_o<'

lib/base.py, Please add the XMLRPCController import

lib/base.py:from pylons.controllers import WSGIController, XMLRPCController

Then, we are setted up to continue with tests

First of all, the doctest boilerplate:

tests/test_doctest_files.py

import doctest
from doctest import DocFileSuite

from myproject.tests import setUp, tearDown
flags = (doctest.ELLIPSIS | doctest.NORMALIZE_WHITESPACE | doctest.REPORT_ONLY_FIRST_FAILURE)

def test_suite():
    return DocFileSuite(
        "test.txt",     
        setUp = setUp,  
        tearDown = tearDown,
        optionflags = flags 
    )

setUp and tearDown will have a central place as they are intialising the application.

As we can't use paste.fixture.TestApp objects with XMLRPC because it does not bind everywhere, the idea is

• launch the server somewhere in a thread
  
• use it later, as usual throught xmlrplib.
  
• We will even declare it as a global to ease the doctests writings.
• We also add a wrapper to url_for to return the host to bind to.

import os
import sys
import re
import threading
from ConfigParser import ConfigParser
from unittest import TestCase

import paste.fixture
import paste.script.appinstall
from paste.deploy import loadapp
from paste.httpserver import serve
from routes.util import url_for

here_dir = os.path.dirname(os.path.abspath(__file__))
conf_dir = os.path.dirname(os.path.dirname(here_dir))
test_file = os.path.join(here_dir, 'test.ini')

cmd = paste.script.appinstall.SetupCommand('setup-app')

cmd.run([test_file])

def setUp(test, *args, **kwargs):
    print "\t-----------------------------------------------------------------"
    print "\t---    Setting up database test environment, please stand by. ---"
    print "\t-----------------------------------------------------------------"
    config = ConfigParser()
    config.read(
        os.path.join(os.path.dirname(sys.argv[0]), '..', 'etc', 'config.ini')
    )

    infos = ConfigParser()
    infos.read(test_file)
    sinfos = infos._sections['server:main']
    wsgiapp = loadapp('config:test.ini', relative_to = here_dir)
    server = test.globs['server'] = serve(wsgiapp,





    t = threading.Thread(target=server.serve_forever)
    t.setDaemon(True)
    t.start()
    test.globs['app'] = paste.fixture.TestApp(wsgiapp)
    def url_for_wrapper(*args, **kwargs):
        lkwargs = {'protocol': 'http' ,'host':  "%s:%s" % (server.server_name, server.server_port)}
        lkwargs.update(kwargs)
        return url_for(*args, **lkwargs)
    test.globs['url_for'] = url_for_wrapper
    test.globs['url_for_orig'] = url_for

def tearDown(test):
    test.globs['server'].server_close()

class TestController(TestCase):
    def __init__(self, *args, **kwargs):
        wsgiapp = loadapp('config:test.ini', relative_to = here_dir)
        self.app = paste.fixture.TestApp(wsgiapp)
        TestCase.__init__(self, *args, **kwargs)

And finally, letz play with our doctest

tests/text.txt

>>> create_url = url_for(controller='mycontroller')

>>> import xmlrpclib
>>> s = xmlrpclib.Server(create_url)
>>> s.index()
'\\_o<'

What i had done on minitage this week end:

Minitage.paste

• Improved varnish configuration
• A new plone profile to be directly mapped to a zeo instance

Minitage.recipe

• Pyc are now just regenerated when eggs are moved into the cache. Thanks to Jim Fulton and the appropriate zc.buildout's function i have borrowed :p
• easy_install method was refactored a bit.
• eggs install method code is now rock solid until the next bug.
  

Thoughts

• For a long time, i think to cache authenticated content though a http proxy-cache. This can be done with the coordination of the plone cachesetup product and varnish. This will be certainly a next improvment to the varnish template. As i will study a cache solution, next week, in my daly work, it will be normally quickly updated :)