When having trouble while installing an egg during the installation phase(easy_install time)

Now the recipe can display some information about which wanted this distribution.

This is even more useful that it support up to 6 dependencies levels from direct to parents !

Here is an example:

Processing ssl-for-setuptools-1.10.tar.gz
Unpacking ssl-for-setuptools-1.10 to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10
Unpacking ssl-for-setuptools-1.10/PKG-INFO to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/PKG-INFO
Unpacking ssl-for-setuptools-1.10/setup.py to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/setup.py
Unpacking ssl-for-setuptools-1.10/ssl to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/ssl
Unpacking ssl-for-setuptools-1.10/ssl/2.3.6 to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/ssl/2.3.6
Unpacking ssl-for-setuptools-1.10/ssl/2.3.6/socketmodule.h to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/ssl/2.3.6/socketmodule.h
Unpacking ssl-for-setuptools-1.10/ssl/2.5.1 to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/ssl/2.5.1
Unpacking ssl-for-setuptools-1.10/ssl/2.5.1/socketmodule.h to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/ssl/2.5.1/socketmodule.h
Unpacking ssl-for-setuptools-1.10/ssl/__init__.py to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/ssl/__init__.py
Unpacking ssl-for-setuptools-1.10/ssl/_ssl2.c to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/ssl/_ssl2.c
Unpacking ssl-for-setuptools-1.10/test to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/test
Unpacking ssl-for-setuptools-1.10/test/badcert.pem to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/test/badcert.pem
Unpacking ssl-for-setuptools-1.10/test/badkey.pem to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/test/badkey.pem
Unpacking ssl-for-setuptools-1.10/test/https_svn_python_org_root.pem to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/test/https_svn_python_org_root.pem
Unpacking ssl-for-setuptools-1.10/test/keycert.pem to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/test/keycert.pem
Unpacking ssl-for-setuptools-1.10/test/nullcert.pem to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/test/nullcert.pem
Unpacking ssl-for-setuptools-1.10/test/test_ssl.py to /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/test/test_ssl.py
Running ssl-for-setuptools-1.10/setup.py bdist_egg --dist-dir /tmp/easy_install-Svj7d6/ssl-for-setuptools-1.10/egg-dist-tmp-NfSQin
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 1714, in main
    with_ei_usage(lambda:
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 1695, in with_ei_usage
    return f()
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 1718, in <lambda>
    distclass=DistributionWithoutHelpCommands, **kw
  File "/usr/lib/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 236, in run
    self.easy_install(spec, not self.no_deps)
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 452, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 501, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 680, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 957, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/command/easy_install.py", line 946, in run_setup
    run_setup(setup_script, args)
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/sandbox.py", line 29, in run_setup
    lambda: execfile(
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/sandbox.py", line 70, in run
    return func()
  File "/home/kiorky/minitage/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/setuptools/sandbox.py", line 31, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 11, in <module>
    def read(rnames):
ValueError: This extension should not be used with Python 2.6 or later (already built in), and has not been tested with Python 2.3.4 or earlier.
    - Failed specs: ssl-for-setuptools==1.10
    - required by:
        - [Requirement.parse('zc.ssl==1.1')]
    - required by:
        - [Requirement.parse('zc.authorizedotnet==1.3')]
    - required by:
        - [Requirement.parse('easyshop.core==0.1a1')]
    - Failed specs: ssl-for-setuptools==1.10
    - required by:
        - [Requirement.parse('zc.ssl==1.1')]
    - required by:
        - [Requirement.parse('zc.authorizedotnet==1.3')]
    - required by:
        - [Requirement.parse('easyshop.core==0.1a1')]
While:
  Installing zopepy.

We see at first shot that easyshop give us trouble ! Without, its hard to know which distribution want ssl-for-setuptools :)

While upgrading my gentoo based laptop after 8monthes of lazy abandonness in profit of exiting projects, i saw that the gentoo's python was forced to use UCS==4.

What an heck while dealing with the 'minitage env' file which mix the system and project environment resulting in a mixin of the system and project python.

Normally, there are no problem, unless your pythons come with different UCS flavors.

Cool thing is that those errors are not silent, and you see them if you are hitted by this flaw :

    ImportError: /bar.egg/module/_foo.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8
or  
    ImportError: /bar.egg/module/_foo.so: undefined symbol: PyUnicodeUCS2_DecodeUTF8

Making some searches showed me that upstream as in python-dev supports only UCS2 by default (issue discussed on their mailing around 2008) but all distros i am aware of package their distros with UCS==4.

For now, I prefered to stick with upstream (UCS==2), but i think that for the user experience, UCS==4 will be better.

Sad thing is that for existing minitage installations if you want to rebuild your python, do also that:

rm -rf minitage/eggs/cache/*-Major.minor*egg
rm -rf minitage/eggs/*/.installed.cfg

To let minitage rebuild any stuff using UCS2.

See http://en.wikipedia.org/wiki/Universal_Character_Set for reference on UCS.

I refreshed a little the ohloh minitage's related page to have some marketing analysis around minitage basecode.

For this, i wanted to add all repositories, what a heck for 200 repos...

Time to create minitage.ohloh to create the repositories for me using some fake browser ;-).

This package uses lxml (XPath), zope.testbrowser (fake browser) and urllib (some handmade http requests) to make the work done.

with more than 100 repos created :D !

 And sorry ohloh Team for flooding ohloh.net :p

.

Calling strftime on a date/datetime instance on py24/py26 will raise a value error like this:

>>> from datetime import date
>>> date(1800,1,1).strftime('%d%m%Y')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: year=1800 is before 1900; the datetime strftime() methods require year >= 1900

Quite annoying heh ?

The work here was to integrate the working patch in the bugreport to py24 and py26 python packages which are in minitage.

So now, with minitage's pythons, you can work with very old dates by default !!!

Just issue:

virtualenv --no-site-packages --distribute minitage
source minitage/bin/activate
minimerge -s
minimerge -v python-2.6 (or 2.4)

And you can use the following interpreter which support those old ages:

minitage/python-2.6/parts/part/bin/python

And of course the related patch does not break anything:

$ svn info
URL : http://svn.python.org/projects/python/branches/release24-maint
Révision : 75208
$../parts/part/bin/python Lib/test/test_datetime.py
...
----------------------------------------------------------------------
Ran 230 tests in 0.858s
OK


$ pwd
/home/kiorky/minitage/dependencies/python-2.6/Python-2.6.4
$../parts/part/bin/python Lib/test/test_datetime.py
...
----------------------------------------------------------------------
Ran 245 tests in 0.791s
OK

GIVE ME ONLY DEPENDENCIES

You can know select to build all dependencies but not the package.

For exemple, to build all python-2.4 dependencies but not python-2.4 itself, you can do:

minimerge -v --only-dependencies python-2.4

BUILD THIS PYTHON PACKAGE FOR THIS PYTHON

For 'eggs' packages, the default was to make come in dependencies all python avaiable in minitage and to build the 'site-packages-Major.Minor' part if it was a buildout build..

It is know rearranged to be a little more subtile.

Look the following:

minimerge -v mypythonpackage # will be mypythonpackage against all python as usual
minimerge -v mypythonpackage python-2.4# will be mypythonpackage against only python-2.4

Just a little post to say that snowleopard let run minitage.

As a note, you must have all SDK installed, specially the 10.4!

Now, nginx and apache are packaged inside minitage.

You have :

• two templates initiating instance of both webservers in minitage.paste.extras
• the related packages to be build via minitage (nginx-0.8, apache-2.2)

Enjoy running:

easy_install -U minitage.paste
paster create -t minitage.instances.nginx myproject #deploy some nginx instance (init scripts, logrotates, docroot)
paster create -t minitage.instances.apacher myproject #deploy some nginx instance (init scripts, logrotates, docroot)

I have added recently the support for standalone tomcat instances and running applications in those tomcat instances.

For example, to initiate a great hudson (contineous integration), you can run the following:

If you are not inside a minitage environment:

easy_install -U minitage.paste.extras
paster create -t minitage.instances.tomcat myproject # it will ask you for the location of a tomcat extracted tarball
paster create -t minitage.instances.hudson myproject

And if you are not inside a minitage environment, to deploy in a myproject subdirectory

easy_install -U minitage.paste.extras
paster create -t minitage.instances.tomcat myproject inside_minitage=no 
paster create -t minitage.instances.hudson myproject inside_minitage=no

Requirements:

• A tomcat tarball extracted somewhere (CATALINA_BASE for knowsers)
  
• A java installed somewhere with a JAVA_HOME environment variable pointing to it
  

Making more and more complicated instances from server dependencies installed into the minitage tree make me relying on somehow intrusive dependencies.

 As i want minitage.paste to depend on barely nothing else that paster and the minitage core,  i have splitted such 'instances' in an additionnal egg.

 minitage.paste.extras was born.

This package borrow 2 instances at the moment:

• minitage.instances.cas: Install a CAS server
• minitage.instances.openldap: Install an openldap server

Those instances make some nice things as generating on the flag SSL certificates for enabling secured connections to your instance(s).

Thanks to my lovely minitage user remarks, i make some frenglish naming with 'profils' in minitage.paste.

Profils have been renamed to instances as it is more relevant.

paster create -t minitage.profils.env project

become:

paster create -t minitage.instances.env project

Lately, i ran and port problematic packages under FreeBSD. All the packages now seem to install without particular problems.

The tested platform is a FreeBSD-7.1 RELEASE 64 bits.

In the the last weeks, I carried out some changes to minitage.paste.

Now, there are some new profils out there:

• minitage.profils.mysql : a profil to initiate a mysql database right into your project.
• minitage.profils.openldap : a profil to initiate an openldap instance right into your project.
• minitage.profils.pasteinitd: a profil to generate an init script to launch a server configured via Pythonpaste configuration
  

There also some changes on the postgresql profile.

Now:

• You have your system user admin of the database. But you can connect only via localhost.
• The user created via the paster is only administrator on the database, but not on the database server. He can connect from localhost without password, but he must authenticate from other locations.

There is pretty much a long time i wanted to switch my stuff on the BSD License.

More as an habit than a conviction, my source files where header pollutated by the GPL License.

For me, and in my opinion, the GPL is not as free as it claims to.

Why ? Because the final user cannot do what he want with your code.

Some debate will qualify that as communism  (GPL) vs anarchy (BSD). I prefer anarchy :).

Of course i use the "modified BSD license" which is compatible with the GPL.

Some on the concerned:

• minitage.core
• dj.paste
• minitage.recipe
• minitage.paste
• cluereleasemanager.paste
  

As part of a new project deployment plan, we needed to deploy mapscript.

Python-MapScript are the python bindings to the underlying mapserver library.

Where the pain begins is that there is no egg available for it, but an old fashioned distutils distribution;.

My bits there were to make some quickly egg compatible on most unixes. My will is to make it nicer and be integrated on the mapserver trunk code.

You can find the code there : http://git.minitage.org/git/others/mapscript/

And play with the egg:

easy_install mapscript 

You must know that mapserver-config must be available in your $PATH.

It is now an hassle to build all the minitage dependencies by hand.

Naturally, the idea to set up some tinderbox like buidbot comes to my mind.

More over, i had a though of generating some binaries to avoid the compilation time for users who don't want to/can't compile.

The lead was to use collective.buildbot, a great recipe to integrate buildbot with zc.buildout deployment.

The result is there : http://git.minitage.org/git/minitage/buildbot/

The idea was to generate a "collective.buildbot" project per dependency. To achieve that, i made a script to generate those projects from the current official minilays.

Compilation is triggered via manual activation (irc, web) but also by a cron scheduler (nearly each 3 hours)

I made a patch to the recipe to have multiple hours values (* 1,2,3 * * *) cron entries, see http://git.minitage.org/git/minitage/buildbot/tree/collective.buildbot.multiple.patch

The buildbot can be viewed there : http://www.minitage.org/buildbot/waterfall

Next steps are to finnish the "binaries dealing" in minitage.core and the mirrors setup, and minitage users will be able to use binaries and no compilation anymore :)

Thanks to a workmate, Jean-Philippe, who shared me a little space of it's Osx Leopard based laptop, i could test minitage on Leopard.

I was surprised to not see a lot of breakages. Most of the packages build fine right now.

I was just anooyed with 'MACOSX_DEPLOYMENT_TARGET' bugs i introduced with previous minitage versions, where i didnt know already all the impact of using it.

So, now, is minitage ready for mac ?

I think, Yes as almost everything inside minitage can install without problems. But ..., it needs to be tested a lot more :)

And think, that you must run OSX Leopard. Prior versions of MacOSX wont be supported. Indeed, i need some linker features which need a lot of rewrites of minitage composants  to be compliant with the old mac linker. (-Wl,-rpath -Wl,... -> -rpath something). Whereas, now, a lot can be fixed in one place, the project has history and there is a lot of places to fix and test to rewrite this "flag issue". But if you want, port it ;)

One recommandation if you use minitage on OSX is to use the boostrapper to get the base python compiled with the right options.

If you get some obscure  errors like:

distutils.errors.DistutilsPlatformError: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.3" but "10.5" during configure

use the bootstrapper ;)

 You ll need also xcode and some tools installed, see on the minitage installation documentation : http://minitage.org/installation.html

I finnished ysterday to make the neccessary patches to make compile all dependencies on the minitage tree on linux 64 bits.

The guest OS is a debian etch 64.

There were some trivial fixes, mainly the whole packages compiled without any problems.

One reason among others  why i love gentoo.

• I got some troubles packaging a package on minitage for its x64 port.
• I looked the gentoo's relevant ebuild.
• I found some interresting patch, i integrated it on the libjpeg buildout.
• I launched the minimerge dance, Et voila, bug resolved !

Thx gentoo and thx gentoo developpers :-)

This website just got updated to latest plone available... There may be bugs with quills ... :)

I worked recently on merging my collective.anonymousbrowser back to zc.testbrowser.

It's definitivly seems to run fine with python >= 2.5.

Points of the merge were large:

• Integrate the proxy balancing capability
• Integrate the user agent faking capability
  
• Add tests where it was missing
• Improve the "real" class
  
• Make the real class mozrunner powered, that's free you to configure a firefox, and just let the code configure a default profile for you.
• Integrate the existing "screenshot" javascript function as a class method. Thats enables you to do browser.screenshot() and that makes a screenshot of the running firefox.
  

But, something i hadn't tested, ..., was the python 2.4 support. Both zc.testbrowser and mozrunner have problems with it...

I will try to make the next commits fix that if possible.

As it's not yet officially released., you can try the code there:

   svn://svn.zope.org/repos/main/zc.testbrowser/branches/fastreal_screenshots_mozrunner_proxys

I made for a workmate yet another django paste's http factory.

The goals were to integrate its way to work, with setting some DJANGO_SETTINGS_MODULE key in the environment and launch the "bidule", and letz play.

The challenge was, ..., to be simple or this workmate will never use this stuff. She was used to tell only about django-admin runserver :).

One part of the challenge was to show some interessant things that we can do with paste configurations, like having different but closer configurations for production or development mode.

Thus, including showing the nice "WebError#eval" application where you can debug your messy code though the web with an interactive debugger.

But, right to the topic, the dj.paste stuff:

• It enables you to have multiple tracs in the same pipeline
• You can integrate it in an exiting paste configuration, it's the goal after all :)
• It 's simple to use, an a sample paste configuration can be:
  

[composite:main]
use = egg:Paste#urlmap
/ = foo

[app:foo]
use=egg:dj.paste
django_settings_module=foo.settings

I did not find the time earlier to post about it.

Minitage 1.0 is out for a week now.

What you can find there ?

1. Documentation has been drasticly improved
   See by yourself:
   
   • minitage
   • minitage.recipe
   • buildout.minitagificator
   • dj.paste
   • minitage.paste
     
   • minitage.core
2. Trac got Git support
3. There are cgit and gitweb instances configured against all minitage repositories
   
4. A lot of bugfixes, as always
5. Recipes have been rewritten a bit to stick to zc.recipe.* implementations
6. minitage.recipe egg based recipes got a fresh touch:
• They do not totally rely any more on easy_install dependency system, just on their declared dependencies (extras are also supported). Thas was something i disliked with zc.recipe.egg, potentially, the easy_install call can install something, which we would'nt. And also, in some cases, the extras requirement could be wiped out. All are bugs reported upstream on the zc.buildout launchpad bugtracker.
• Their installer should now try not to look in the site-packages first, but rather the eggs cache to avoid conflict errors.
  
• I also implemented an adaptation of the zc.buildout redo_pyc function which seems to be somehow slow.
  
7. Welcome to the buildout.minitagificator egg which help minitage to conquer the world with replacing some well known recipes with their equivalent minitage implementations with some violent monkey patches. It's a buildout extension.
   
8. The "static fetcher" has been improved and bug fixed
9. There some new cool API functions like a search function which take regexes to look for minibuilds in the available minilays
   
10. minitage.paste got a great step forward with all pastes
11. All templates now have now minitage layout as an optionnal dependency. That means that you do not need minitage to generate templates, however, the normal use is to use them in minitage.
    • Plone32 template:
      • Predefined set of products to install
      • A developper mode where you have aditionnal tools installed (ptprofiler, collective.autorestart on linux, and so on) and debug mode turned on by default
      • wsgi configuration and launchers
      • A working filesystemstorage integration
      • You can use either RelStorage, ZEO or classical ZODB access
        
      • You can give to paster whatever you want to custom the buildout generation by some predefined keys.
    • Zope3 template:
      • wsgi configuration and launchers (paster and Zdaemon based)
        
      • You can use either RelStorage, ZEO or classical ZODB access
      • You can give to paster whatever you want to custom the buildout generation by some predefined keys.
    • Django template:
      • wsgi configuration and launchers
      • GeoDjango support
      • dj.paste (http://pypi.python.org/pypi/dj.paste) integration. This paster factory enables you to have multiple django instances in the same pipeline
        
      • A basic application created when you generate your project to quickstart.
      • You can give to paster whatever you want to custom the buildout generation by some predefined keys.
    • Nearly all templates which have WSGI support got Spawning integration. Spawning is an über performant python based WSGI server.
      
    • postgresql, varnish, varnish2, and env profils:
      • Can run now without minitage or any project installed
    • The new paste-initd profil:
      • it generates an initscript wrapper to trigger a "paster serve" from some paste configuration of your choice

What are the minitage projects and profiles, after all ?

All are paster (see PasteScripts on pypi) templates.

Projects target to quickstart an application or a project like a plone3 or a django website.

Profiles, on the other side, are a piece of integration, targeted to be in general integrated into a minitage based project. That can be a postgresql database instance, or a shell environement file with has all minitage dependencies, that you can source when playing with your project.

I have a crappy hard drive which claims to be UDMA-66 compatible, however, its an old PIO4 based one ...

After successfully upgrading the system to Freebsd 7.1 this week end, at 3.30AM, i can't boot :(

The driver want to go UDMA, the disk doesn't understand.

No mean in the bios to force that... And no more internet because this box is the home router !

What to do ?

Go to use my 3G phone to get my ISP DNS because ... the funny "freebox" is customizable throught the web, on the free.fr website.

You have no local access on it.

So now, i have some route and DNS setted up, i can go to read

man loader(8)
man ata(4)

And now, i know some interresting new things:

Disable dma:

set hw.ata.ata_dma=0

Prompt for root partition when the partition location changed:

set boot_askname

A teamate wanted to have a binary like the package manager he was used to use.

Its done, now instead of calling minimerge, you can use either:

• minimerge
• yasty
• apty
• pacmerge
  

During last weeks, i have switched to git minitage, and by default all come from it.

The failure here is that my git fetcher is not very sympathic with older versions of git (<1.5).

But on debian and red hat legacy, it's a pain in the ass to have relativly new stuff !!!

To resolve that, with workmates, we come to the decision to fetch packages over http.

And now, it's done. Minimerge codebase is 'HttpCompliant' in sense that defaults minilays and packages are fetched via the minitage static fetcher.

This one just use urllib, and you can even set proxies in your minimerge.cfg

To be up-to-date:

easy_install -U minitage.core
mv etc/minimerge.cfg etc/minimerge.cfg.old
cd minilays
mkdir ../old
mv meta misc eggs dependencies ../old
minimerge -s

   This afternoon, Caramelle died from a fulgurant flue after 2 years of happy pouic pouics.

I was going to the vet when she died, and it must have been terrible for her.

And that, one hour after the starting of the symptoms.

R.I.P. Caramelle.

For those who get annonying troubles installing reportlab 2.3 with the
distribution available on pypi, i made another distribution [1].
To achieve this, i have just done a checkout from their svn TAG (2.3) and
re-executed the sdist dance.

The troubles i had personnaly was something like that :

	"/usr/bin/ld: cannot find -l_renderPM_libart"

Here is a sample buildout using either minitage.recipe:egg or zc.recipe.egg to
install it:

[buildout]
versions=versions
parts=
    eggs
    zceggs
[versions]
reportlab=2.3

[eggs]
recipe=minitage.recipe:egg
eggs=
    reportlab
find-links=
    http://distfiles.minitage.org/public/externals/minitage/

[zceggs]
recipe=zc.recipe.egg
eggs=
    reportlab
find-links=
    http://distfiles.minitage.org/public/externals/minitage/

[1] http://distfiles.minitage.org/public/externals/minitage/reportlab-2.3.tar.gz

I recently add some GIS packages (pgrouting) and their dependencies.

Here is a little list of what minitage can offer to you:

• bzip2, zlib
• cmake, scons
  
• libwww, curl, libssh, openssl,
  
• git, subversion, mercurial, bazaar
  
• gdal, geos, pgrouting
  
• postgresql, postgis
• openldap
• python's, PIL, psycopg
  
• libxml, libxslt and expat
• libtiff, libjpeg, libpng, libgif
• etc.

And next to come:

• CGAL
• GAUL
  

As a part of a recent discussion i had on irc with Stephan Richter, one of my next works will be to merge zc.testbrowser with collective.anonymousbrowser.

I worked on collective.anonymousbrowser to allow the use of proxys and proxy balancing over zc.testbrowser proxys.

Another interressant feature fir the project was also to retry urls if there were errors.

I made also some bugfix and improvments on the zc.testbrowser.real.Browser side like using mozrunner to control firefox, loading directly the js file if we are local..

What i want to do now is to port all the interresting things back to zc.testbrowser.

Hope i will have time for that this week.

Just an entry to thank smplayer developers for their great piece of software!

/me just got another subtitle  via the opensubtitles.org smplayer's menu entry.

I really need this feature as i am not already fluent in japanese :(

As a really important part for minitage to conquer the world, minitage paster templates must rock.

So feel free to install and test the brand new zope3 template.

In an activated minitage instance:

easy_install minitage.paste
paster create -t minitage.zope3 z3
minimerge z3
cd zope/z3
bin/z3-ctl fg

Plone32 was written before the recipes rewrite, it should work but i must revisite it. Should be alvalaible tomorow!

If you didnt know minitage.paste, you can also check the other templates and projects

paster create --list-templates|grep minitage

And visit the pypi and minitage.org page :)

To follow zc.buildout new changes, i have decided to make all minitage recipes more "zc.recipes" compliant.

Now, those recipes share the same API and you an use them "really" as the original zc.recipes..

Another iinterresting point is that i never liked the way to compute dependencies.
Another point was to allow easy_install to get automaticly our stuff out.
To correct all that weird, i have rewritten the dependencies calculation not to let easy_install
do anything but install one thing at a time, and with versions and extras checks.

A part of the improvments:

• versions are really fixed, imperative way.
  
• dependencies are better handled
• egg's extras requirements (like something.foo [test, zcml]) are really well installed with all their dependencies
• develop eggs get also their dependencies

You can now check out and test buildout.minitagicator.

This is a little extension for buildout that enables you to override/monkey patch some buildout stuff to use minitage internals instead.

If you like minitage, if you like the minitage recipes, you ll enjoy it!

See http://pypi.python.org/pypi/buildout.minitagificator for more information :)

There some points i have added to minitage since october:

• I moved minilays, dependencies and code on git because mercurial forest extensions really lacks maintenance
• I have done some maintenance over minitage.paste's templates (plone25, env, postgresql for postgresql-8.3) and added some (plone32svn, pylons)
• minitage.recipe received a lot of bugfixes and enhancements like the minitage.recipe:fetch entry point
• minitage.core has also been bugfixed specially for the git wave
• eggs have been buildoutified and testrunnerified
  

Those are the features i ll release on the next days:

• A brand new plone 32 paster
• A revisited zope3 paster
• minitage.recipe:eggs now support develop eggs dependencies installation
• minitage.recipe:eggs now has the same options and API that zc.recipe.egg for their common parts
  
• minitage.recipe:scripts calls minitage.recipe:eggs prior to the script generation, there is no need anymore to have too redundant parts in buildouts
  
• minitage.buildout extension is a buildout extension that monkey patches zc.recipe.egg with minitage.recipe.egg
• the bootstrapper script is refreshed :
• update all sotfwares to lastest versions
  
• includes git-1.6 which is required for minitage to fetch its core packages

TODO as an though for contributors:

• update doc for git changes
• update doc to explain how to move an existing buildout to a minitage compliant one
• documente the recipes
  

I m actually developping some application around XMLRPC protocol at work.

We are using Pylons for the framework part, and i played this afternoon at setting up some testing environnement for doing doctests.

This test is a proof of concept, it 's code extracted from our internal application, it's just a starter for you. The whole is working with some tweaks.

controllers/mycontroller.py, a simple controller doing simple stuff

class MyController(XMLRPCController):
    """controller."""                           

    def index(self):
        return '\_o<'

lib/base.py, Please add the XMLRPCController import

lib/base.py:from pylons.controllers import WSGIController, XMLRPCController

Then, we are setted up to continue with tests

First of all, the doctest boilerplate:

tests/test_doctest_files.py

import doctest
from doctest import DocFileSuite

from myproject.tests import setUp, tearDown
flags = (doctest.ELLIPSIS | doctest.NORMALIZE_WHITESPACE | doctest.REPORT_ONLY_FIRST_FAILURE)

def test_suite():
    return DocFileSuite(
        "test.txt",     
        setUp = setUp,  
        tearDown = tearDown,
        optionflags = flags 
    )

setUp and tearDown will have a central place as they are intialising the application.

As we can't use paste.fixture.TestApp objects with XMLRPC because it does not bind everywhere, the idea is

• launch the server somewhere in a thread
  
• use it later, as usual throught xmlrplib.
  
• We will even declare it as a global to ease the doctests writings.
• We also add a wrapper to url_for to return the host to bind to.

import os
import sys
import re
import threading
from ConfigParser import ConfigParser
from unittest import TestCase

import paste.fixture
import paste.script.appinstall
from paste.deploy import loadapp
from paste.httpserver import serve
from routes.util import url_for

here_dir = os.path.dirname(os.path.abspath(__file__))
conf_dir = os.path.dirname(os.path.dirname(here_dir))
test_file = os.path.join(here_dir, 'test.ini')

cmd = paste.script.appinstall.SetupCommand('setup-app')

cmd.run([test_file])

def setUp(test, *args, **kwargs):
    print "\t-----------------------------------------------------------------"
    print "\t---    Setting up database test environment, please stand by. ---"
    print "\t-----------------------------------------------------------------"
    config = ConfigParser()
    config.read(
        os.path.join(os.path.dirname(sys.argv[0]), '..', 'etc', 'config.ini')
    )

    infos = ConfigParser()
    infos.read(test_file)
    sinfos = infos._sections['server:main']
    wsgiapp = loadapp('config:test.ini', relative_to = here_dir)
    server = test.globs['server'] = serve(wsgiapp,





    t = threading.Thread(target=server.serve_forever)
    t.setDaemon(True)
    t.start()
    test.globs['app'] = paste.fixture.TestApp(wsgiapp)
    def url_for_wrapper(*args, **kwargs):
        lkwargs = {'protocol': 'http' ,'host':  "%s:%s" % (server.server_name, server.server_port)}
        lkwargs.update(kwargs)
        return url_for(*args, **lkwargs)
    test.globs['url_for'] = url_for_wrapper
    test.globs['url_for_orig'] = url_for

def tearDown(test):
    test.globs['server'].server_close()

class TestController(TestCase):
    def __init__(self, *args, **kwargs):
        wsgiapp = loadapp('config:test.ini', relative_to = here_dir)
        self.app = paste.fixture.TestApp(wsgiapp)
        TestCase.__init__(self, *args, **kwargs)

And finally, letz play with our doctest

tests/text.txt

>>> create_url = url_for(controller='mycontroller')

>>> import xmlrpclib
>>> s = xmlrpclib.Server(create_url)
>>> s.index()
'\\_o<'

A little but wonderfull tool in the python world is paster.

It permits you to generate with some code logics a full set of files.

You can for example create a buildout, apache configurations files and etc.

The documentation for profils are in the official minitage documentation : http://www.minitage.org/doc/rst/

What is interesting there is that i use it into minitage to generate stuff inside.

There are 2 main sort of templates:

• Profils: Something relative to a project, a database, a ldap instance (not finnished atm.), a varnish instance... Profils are applied onto an existing project !
  
• Projects: A project and a minibuild referencing to it, a zope project, a turbogears project ...

To get those templates in a minitage instance:

source minitage/bin/activate
easy_install -U minitage.paste
paster create --list-templates

 After you can for example create a plone31 projet:

paster create -t minitage.plone31zeo myproject

 And after that, because plone is slow, put inside a varnish instance

paster create -t minitage.profils.varnish myproject

 Because you need svn 1.5 but it is not packaged on your debian potatoe, just reference it in the dependencies part of your project minibuild:

dependencies=...pilwotk-1.1.6.4 ...
install_method=buildout
src_uri=http://hg.foo.net/myproject
src_type=hg
category=zope
homepage=http://foo.net
description= a plone 3.1 buildout for myproject

And install the minitage.env template:

paster create -t minitage.profils.env myproject

 This profil will create for you a small bash script that will change your environnemnent according to 'myproject' needs. PATH, LD_LIBRARY_PATH and so will be feeded with all minitage dependencies. That will enable you to use svn-1.5 when you source it (like source bin/activate)

source zope/myproject/sys/share/minitage/minitage.env
which svn
/minitage/dependencies/subversion-1.5/parts/part/bin/svn

Another interresting integrations are the zope3 and tubogears projects.

• Zope3 template realisation allowed me to make some fixes in the minitage.recipe:scripts recipe. Now, this one generates for you great launchers scripts which include ALL their respective dependencies.
  
• Something particular with tubogears is that i must have done some overhead over buildout.

You can see the full documentation there : http://www.minitage.org/doc/rst/tg_project.html

Testing the 'serve-branches' script, as mwh asked me on the previous bug report, i have now something pretty cool :)

So, something nice is that i have dropped away the ugly patch.

see : http://bzr.cryptelium.net in action.

What's next?

Michael Hudson, the main developper, told me that the idea is to drop away the loggerhead.conf and to use serve-branches as the main thing to use.

That's a good idea. Usage is simple, nothing to worry about configuration, it just does publishing after all.

Little bits i can see as improvments (always reported/planned):

• a breadcrumb in repo view to browse back
• Maybe some sort of ajaxified navigator to have the whole tree from the homepage
  

What i had done on minitage this week end:

Minitage.paste

• Improved varnish configuration
• A new plone profile to be directly mapped to a zeo instance

Minitage.recipe

• Pyc are now just regenerated when eggs are moved into the cache. Thanks to Jim Fulton and the appropriate zc.buildout's function i have borrowed :p
• easy_install method was refactored a bit.
• eggs install method code is now rock solid until the next bug.
  

Thoughts

• For a long time, i think to cache authenticated content though a http proxy-cache. This can be done with the coordination of the plone cachesetup product and varnish. This will be certainly a next improvment to the varnish template. As i will study a cache solution, next week, in my daly work, it will be normally quickly updated :)
  

Althought i was playing in the wonderfull world of mercurial, i 'm trying out bazaar and its tools.

The goal of the week end was to set up some repositories and a source browser on top of them.

In bazaar world, there is not much choice to achieve that. In other terms, you re really encouraged to use loggerhead.

So, i began to set up a repository, file access and so on... Then look at the loggerhead documentation. Great, there is some sort of auto publishing feature.

This is a mechanism like our beloved mercurial's hgwebdir.cgi to autopublish branches inside a top level directory.

Installation

So, the procedure i used was really simple:

Grab it

bzr branch http://bazaar.launchpad.net/%7Eloggerhead-team/loggerhead/trunk/ loggerhead.dev

Edit and point to my top level directory

cp loggerhead.conf* loggerhead.conf

vi loggerhead.conf

Set up a gentoo init script to start it and schedule for starting after apache2

$ cat /etc/init.d/loggerhead
#!/sbin/runscript
# Copyright 1999-2004 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

depend() {
        need net apache2
}

start() {
        ebegin "Starting $server_name server"
        start-stop-daemon --start  --user $user  --quiet --background \
        -p $pid_file -m  --exec "$py"  --  "$start_cmd"
        eend $?
}

stop() {
        ebegin "Stopping $server_name  server"
        "$py"  --  "$stop_cmd"  2>&1 >> /dev/null
        eend $?
}

$ cat /etc/conf.d/loggerhead
start_cmd=/path/loggerhead.dev/start-loggerhead
stop_cmd=/path/loggerhead.dev/stop-loggerhead
pid_file="/path/loggerhead.dev/loggerhead.pid"
user="apache"
py=`which python`
server_name="Loggerhead"

 Configure apache for serving both bazaar and loggerhead on the same vhost

...
Vhost basic and directories directives
...
# setting an alias to point to the bazaar directory

Alias /repos/ /path/repos/


RewriteEngine on

# mapping to /repos  if we are not browsing on loggerhead or if we are directly "branching"
RewriteCond  %{REQUEST_URI}  !^(/repos/.*)$
RewriteCond  %{REQUEST_URI}  ^(.*\.bzr.*)$
RewriteRule  ^/(.*)$         /repos/$1   [R,L]

# mapping to loggerhead otherwise
RewriteCond  %{REQUEST_URI}  !^(.*\.bzr.*)$
RewriteCond  %{REQUEST_URI}  !^(/repos/.*)$
RewriteCond  %{REQUEST_URI}  !^(/icons/.*)$
RewriteRule  ^/(.*)          http://bzr.cryptelium.net:1337/$1 [P]
RewriteCond  %{REQUEST_URI}  !^(.*\.bzr.*)$
RewriteCond  %{REQUEST_URI}  !^(/repos/.*)$
RewriteCond  %{REQUEST_URI}  !^(/icons/.*)$
RewriteRule  ^/(.*)/$        http://bzr.cryptelium.net:1337/$1 [P]

# The two [P] RewriteRules are just ProxyPass rules which allow to redirect to the loggerhead server.

Are we done yet ?

Unfortunatly not.

Loggerhead is stupid and will not publish two branches with same name.

Another thing is that it will take the branch name for display in UI.

I REALLLY do not like that. I prefer to see full pathes in the UI. That helps my little brain with tree view focus to function properly.

Why ?

This cannot be served (just b/c will be):

a/c
b/c

b/c/d

Results in

d

Fix ?

For my own use, i make my own fix :)

But i upstreamed the result there:https://bugs.launchpad.net/loggerhead/+bug/254411

This allows me to publish my repositories with path views :)

Result

http://bzr.cryptelium.net/

There is a long time i must have done that.

IT s done now, my blog is open :)

Stay tuned for new entries.

What will may be next:

• Plone customization
• Refactor logo and design of this bare plone.
  
• Integration with cryptelium services
• Cofee and tea