2010-09-17

Installing Multiple Perls with App::perlbrew and App::cpanminus

Having learnt from my previous mistakes, this is a simplistic way to set up multiple somewhat isolated installs of Perl in a user profile

App::perlbrew is a very handy tool for managing several user-installs of Perl, and facilitates the easy switching between Perl versions.

App::cpanminus is the most straight-forward and lightweight cpan client I've ever seen, and it just works, and works well, and leads to relatively pain-free installation 80% of the time.

1. Install A Bare copy of Perlbrew

Getting a copy of Perlbrew should be the very first thing you do. No cpanm, no local::lib, just straight perlbrew.
$ cd ~ 
$ curl -LO http://xrl.us/perlbrew

2. Setup Perlbrew

Once we have a copy of perlbrew, we run the install command of it, which completes the bootstrapping of perlbrew. Then all thats needed is to update your profile with the right magic line so that new shells will have the right environment set up.
$ perl ~/perlbrew install
$ rm ~/perlbrew
$ ~/perl5/perlbrew/bin/perlbrew init
# use the line perlbrew spits out.
$ echo "source /home/test_1/perl5/perlbrew/etc/bashrc" | tee -a ~/.bashrc

3. Enter your new perlbrew ENV

Now we enter our new shell so that we can test the change to our configuration. We run env and grep the PATH value just to double check perlbrew has worked properly.
$ bash
$ env | grep PATH
PATH=/home/test_1/perl5/perlbrew/bin:/home/test_1/perl5/perlbrew/perls/current/bin:/home/test_1/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin:/usr/games:.

4. Choose a mirror

This step is mostly optional, but it lets you choose which mirror perlbrew will download Perl sources from, so a local one is best for speed sakes.
$ perlbrew mirror

5. Install your wanted perl versions

Now we perform the slow installation of our Perls. In my case, I'm installing a copy of the current stable ( 5.12.2 ) and the current development release ( 5.13.2 ). The -v is optional, but you'll want it if you do not wish to die of boredom because it generally just sits there doing nothing for 10+ minutes without it.
$ perlbrew -v install perl-5.12.2
$ perlbrew -v install perl-5.13.4

6. Setup 'cpanm' for each perl

This step appears to be the most important step. If you previously had cpanm installed with system perl you do NOT want to be using that at all. When cpanm is installed, the bin/ script hard-codes a path to the perl it was installed with, so using cpanm built with system perl will build installed modules using that system perl instead, and using its install paths and soforth, and you do not want this. So, you must install a cpanm for each perl using this bootstrap technique.
$ perlbrew switch perl-5.12.2
$ curl -L http://cpanmin.us | perl - App::cpanminus
$ perlbrew switch perl-5.13.4
$ curl -L http://cpanmin.us | perl - App::cpanminus

7. Configure local cpans

Strangely, I've found a few modules I try install tend to expect a working CPAN install, regardless of what tool I'm actually using. This should be fixed, but there is a practical work-around until then. Simply configure cpan!
$ perlbrew switch perl-5.12.2
$ cpan
# Answer all setup instructions
» o conf commit
» q
$ perlbrew switch perl-5.13.4
$ cpan
# Answer all setup instructions
» o conf commit
» q

8. Test your installs

This is a list of things I've found to trip up various corner cases and indicate you've built it wrong.
$ perlbrew switch perl-5.12.2
$ cpanm --interactive -v App::cpanoutdated
$ cpan-outdated
$ cpanm --interactive -v App::CPAN::Fresh

$ perlbrew switch perl-5.13.4
$ cpanm --interactive -v App::cpanoutdated
$ cpan-outdated
$ cpanm --interactive -v App::CPAN::Fresh
With all things going to plan, those 2 things at least should build and be runnable. cpan-outdated and cpanf should both be runnable in both perls without complaining it cant find their modules, and CPAN::Inject and Compress::BZip2 should install without strange failures. ( those 2 modules lead me in prior cases to discover broken setups that needed fixing to work, so hopefully, going to plan, following the instructions above will avoid this havoc. )

9. Profit!

Thats all there is to it. Note we do NOT use local::lib for this setup. Using each Perls default local module installation directory should be perfectly satisfactory, and as long as you're in a properly configured ENV and you're using 'perlbrew' to select perl's that are not system perl, everything should be sweet =).
Ok, lots of things on my machine fail to build still, but those peskynesses I'm convinced are unrelated to the Perl setup.

10. Credit

Props to the people who helped me out with working out this configuration ( brian d foy, miyagawa, John Napiorkowski ) and to the authors of cpanm ( miyagawa ) and perlbrew ( gugod ). These are awesome tools, and once you learn them, they really can make working with Perl a much more pleasureable experience!. And also props and ♥ to the Perl Community for simply existing, and fostering this development path.

2010-09-15

I ♥ the Perl Community

miyagawa++,jjnapiork++,bdfoy++.

Perl is awesome, but the community is better, with nothing even in competition as I know it.

Where else can you blog about a confusing corner case you hit in a seemingly rare operating system and get Excellent answers from not only great people, but the author of the module the problem was in, one of the people who wrote or contributed a lot of the other useful tools you use, and the author many recognized Perl books

And then, not only did I get the right solution for my problem, but many other alternative good approaches, as well as answering parts of the question I didn't even ask with side tips that seem "related enough" that I'd likely encounter in similar ways, and how to make my life easier when that happens.

I ♥ this positive approach to programming, where people are not only caring about solving my specific problem, but suggesting things that can help me become a better programmer as a whole, and I'm frankly proud simply to be involved with a community which has such a valuable work-ethic.

Frankly, its a shame its so hard to sell Perl on the community aspect, because it is just awesome in ways I've never seen before in a Programming Language, and it by far trumps technical aspects in terms of awesomeness. If brainf**k had a community as awesome as Perl has, it would probably be better than many languages simply because of the community aspect, at least in my opinion. Its just a shame you can't convey how great such a community being preset is to newbies to the language without first immersing them in the culture and community, because to understand and appreciate, I think you must first experience it.

OpenBSD + Perl + Modern Tools and Approaches -> Me = Confused :(

So, I'm doing my first attempt at a hand-holding free install of Perl. I'm used to the niceties of Gentoo and being able to do everything through its package manager, so I thought I'd try doing it the way everyone else in the world apparently uses as "The most practical".

I'm going to walk you through what I did, mostly constructed from memory, so you have an idea of what the problem I have is, or, if you're in a similar situation, you can get some progress and learn from my mistakes once I've worked out what I need.

Normally, I'd ask about this on #perl@irc.freenode.org or something on irc.perl.org, or if appropriate, file a bug. However, in this case, I can't even conceive of which would be the right place to target my question, OpenBSD is in my estimation very "niché" market at the moment, as are lots of the modern tools for Perl, and I don't know where the appropriate place to solicit help for them are. So, I approach the ALL MIGHTY LAZY-WEB.

The Setup

  1. Installed OpenBSD 4.7
    This shall be left as an exercise to the reader as to how this works.  Its too much to cover here, and it really is pretty straight forward =).
  2. Install cpanm
    Everyone I see in Perl these days seems to be ranting about this, so I used the perscribed instructions:    
    $ curl -L http://cpanmin.us | perl - --sudo App::cpanminus
  3. I don't want to be stuck using Perl 5.10.1, which is great and all, but I'd rather be doing work with 5.12.2 and 5.13.* . And I keep getting recommendations NOT to use system Perl for ANYTHING other than getting your custom Perl running. ( Using system Perl is fine in Gentoo, at least how I use it, we've got 5.12.2 in tree now, and stuffing Perl dists into Package Management JustWorks™ ). The new sex for this is allegedly perlbrew, so I'm firing that baby up next.
    $ cpanm --sudo App::perlbrew
  4. All appears good!. Now from here on, is where I think a few things start to drift south, but not entirely sure WHERE.
    $ perlbrew init
    # add instructed line to bash
    $ bash
    $ perlbrew install perl-5.13.4 -v
    $ perlbrew install perl-5.12.2 -v
  5. All this appears to run smoothly.
    $ perlbrew switch perl-5.13.4
  6. Here is where I do the stupid things that possibly lead to my downfall. First, you must understand how I want my setup:
    1. I want my primary development user (kent) to have 2 copies of Perl available, 5.13.4 and 5.12.4
    2. I want the modules for each install of Perl to follow their respective installs so I can just switch between Perls and have the modules switch over too
    3. "Production" Will repeat this process, except with less versions of Perl, and probably with less modules installed.

    To achieve this, I insert lines in my .bashrc until it resembles this
    source /home/kent/perl5/perlbrew/etc/bashrc
    export PERLDIR=/home/kent/perl5/perlbrew/perls/current
    export MODULEBUILDRC=/home/kent/perl5/perlbrew/etc/.modulebuildrc
    export PERL_MM_OPT="INSTALL_BASE=${PERLDIR}"
    export PERL5LIB="${PERLDIR}:${PERLDIR}/i386-openbsd"
    export PERL_CPANM_OPT="--local-lib=${PERLDIR}"
    
    and .modulebuildrc of course contains this:
    install  --install_base  /home/kent/perl5/perlbrew/perls/current/
    
  7. For the most part this works perfectly, and I'm off installing modules happy as Larry.
  8. And then a few hours later, something depends on IO::Compress::BZip2. Now is the beginning of sorrows.

The Problem:

Can't find libbz2!

I'm sure as eggs I have bzip2 and family installed and working.
However, this worrisome notice appears during build:
 Entering Compress-Bzip2-2.09
Configuring Compress-Bzip2-2.09 ... Running Makefile.PL
Parsing config.in...
/usr/bin/ld: cannot find -lbz2
collect2: ld returned 1 exit status
compile command 'cc -fno-strict-aliasing -fno-delete-null-pointer-checks -pipe -fstack-protector -I/usr/local/include -Wl,-E  -fstack-protector -o show_bzversion show_bzversion.c -lbz2' failed
system bzip2 not found, building internal libbz2
Ah .... ok.
$ bzip2 -h 2❭&1 | head -n 1
bzip2, a block-sorting file compressor.  Version 1.0.5, 10-Dec-2007.
$ /usr/bin/ldd $(which bzip2)
/usr/local/bin/bzip2:
        Start    End      Type Open Ref GrpRef Name
        1c000000 3c006000 exe  1    0   0      /usr/local/bin/bzip2
        065b5000 265b9000 rlib 0    1   0      /usr/local/lib/libbz2.so.10.4
        07295000 272ce000 rlib 0    1   0      /usr/lib/libc.so.53.1
        0643c000 0643c000 rtld 0    1   0      /usr/libexec/ld.so
Ok, so maybe it is a bit geriatric
That should be fine though right? WRONG

Something magical keeps finding Perl 5.10.1 :(

Surely, this abomination will not end well:
Building and testing Compress-Bzip2-2.09 for Compress::Bzip2 ... cp lib/Compress/Bzip2.pm blib/lib/Compress/Bzip2.pm
AutoSplitting blib/lib/Compress/Bzip2.pm (blib/lib/auto/Compress/Bzip2)
cd bzlib-src && make 
cc -c    -fno-strict-aliasing -fno-delete-null-pointer-checks -pipe -fstack-protector -I/usr/local/include -O2     -DVERSION=\"\"  -DXS_VERSION=\"\" -DPIC -fPIC "-I/usr/libdata/perl5/i386-openbsd/5.10.1/CORE"   blocksort.c
cc -c    -fno-strict-aliasing -fno-delete-null-pointer-checks -pipe -fstack-protector -I/usr/local/include -O2     -DVERSION=\"\"  -DXS_VERSION=\"\" -DPIC -fPIC "-I/usr/libdata/perl5/i386-openbsd/5.10.1/CORE"   huffman.c
cc -c    -fno-strict-aliasing -fno-delete-null-pointer-checks -pipe -fstack-protector -I/usr/local/include -O2     -DVERSION=\"\"  -DXS_VERSION=\"\" -DPIC -fPIC "-I/usr/libdata/perl5/i386-openbsd/5.10.1/CORE"   crctable.c
cc -c    -fno-strict-aliasing -fno-delete-null-pointer-checks -pipe -fstack-protector -I/usr/local/include -O2     -DVERSION=\"\"  -DXS_VERSION=\"\" -DPIC -fPIC "-I/usr/libdata/perl5/i386-openbsd/5.10.1/CORE"   randtable.c
cc -c    -fno-strict-aliasing -fno-delete-null-pointer-checks -pipe -fstack-protector -I/usr/local/include -O2     -DVERSION=\"\"  -DXS_VERSION=\"\" -DPIC -fPIC "-I/usr/libdata/perl5/i386-openbsd/5.10.1/CORE"   compress.c
cc -c    -fno-strict-aliasing -fno-delete-null-pointer-checks -pipe -fstack-protector -I/usr/local/include -O2     -DVERSION=\"\"  -DXS_VERSION=\"\" -DPIC -fPIC "-I/usr/libdata/perl5/i386-openbsd/5.10.1/CORE"   decompress.c
cc -c    -fno-strict-aliasing -fno-delete-null-pointer-checks -pipe -fstack-protector -I/usr/local/include -O2     -DVERSION=\"\"  -DXS_VERSION=\"\" -DPIC -fPIC "-I/usr/libdata/perl5/i386-openbsd/5.10.1/CORE"   bzlib.c
ar cr libbz2.a  && ranlib libbz2.a
cc -c    -fno-strict-aliasing -fno-delete-null-pointer-checks -pipe -fstack-protector -I/usr/local/include -O2     -DVERSION=\"\"  -DXS_VERSION=\"\" -DPIC -fPIC "-I/usr/libdata/perl5/i386-openbsd/5.10.1/CORE"   bzip2.c
/usr/bin/perl /usr/libdata/perl5/ExtUtils/xsubpp  -typemap /usr/libdata/perl5/ExtUtils/typemap -typemap typemap  Bzip2.xs > Bzip2.xsc && mv Bzip2.xsc Bzip2.c
cc -c  -Ibzlib-src  -fno-strict-aliasing -fno-delete-null-pointer-checks -pipe -fstack-protector -I/usr/local/include -O2     -DVERSION=\"2.09\"  -DXS_VERSION=\"2.09\" -DPIC -fPIC "-I/usr/libdata/perl5/i386-openbsd/5.10.1/CORE"   Bzip2.c
In file included from Bzip2.xs:7:
ppport.h:231:1: warning: "PERL_UNUSED_DECL" redefined
In file included from Bzip2.xs:4:
/usr/libdata/perl5/i386-openbsd/5.10.1/CORE/perl.h:330:1: warning: this is the location of the previous definition
Running Mkbootstrap for Compress::Bzip2 ()
Um. Um. Um.
How about NO
$ perl -v  | grep version 
This is perl 5, version 13, subversion 4 (v5.13.4) built for OpenBSD.i386-openbsd
That's going to go down like a houseboat on fire.

What comes next is only a natural

t/010-useability.t ...... 1/3 ol 'BZ2_bzDecompressInit'nm/work/1284524774.31144/Compress-Bzip2-2.09/blib/arch/auto/Compress/Bzip2/Bzip2.so: undefined symb
/usr/bin/perl:/home/kent/.cpanm/work/1284524774.31144/Compress-Bzip2-2.09/blib/arch/auto/Compress/Bzip2/Bzip2.so: undefined symbol 'BZ2_bzDecompress'
/usr/bin/perl:/home/kent/.cpanm/work/1284524774.31144/Compress-Bzip2-2.09/blib/arch/auto/Compress/Bzip2/Bzip2.so: undefined symbol 'BZ2_bzBuffToBuffDecompress'
/usr/bin/perl:/home/kent/.cpanm/work/1284524774.31144/Compress-Bzip2-2.09/blib/arch/auto/Compress/Bzip2/Bzip2.so: undefined symbol 'BZ2_bzDecompressEnd'
/usr/bin/perl:/home/kent/.cpanm/work/1284524774.31144/Compress-Bzip2-2.09/blib/arch/auto/Compress/Bzip2/Bzip2.so: undefined symbol 'BZ2_bzCompress'
/usr/bin/perl:/home/kent/.cpanm/work/1284524774.31144/Compress-Bzip2-2.09/blib/arch/auto/Compress/Bzip2/Bzip2.so: undefined symbol 'BZ2_bzBuffToBuffCompress'
/usr/bin/perl:/home/kent/.cpanm/work/1284524774.31144/Compress-Bzip2-2.09/blib/arch/auto/Compress/Bzip2/Bzip2.so: undefined symbol 'BZ2_bzlibVersion'
/usr/bin/perl:/home/kent/.cpanm/work/1284524774.31144/Compress-Bzip2-2.09/blib/arch/auto/Compress/Bzip2/Bzip2.so: undefined symbol 'BZ2_bzCompressInit'
/usr/bin/perl:/home/kent/.cpanm/work/1284524774.31144/Compress-Bzip2-2.09/blib/arch/auto/Compress/Bzip2/Bzip2.so: undefined symbol 'BZ2_bzCompressEnd'
And more and more of that explosion until you see:
Files=25, Tests=33,  7 wallclock secs ( 0.35 usr  0.21 sys +  4.74 cusr  1.44 csys =  6.74 CPU)
Result: FAIL
Failed 25/25 test programs. 30/33 subtests failed.
Oh crap. That's not good.
Something Seriously wrong is going on here, but hell knows what it is, and I'm the least qualified to work it out.

Call For Halp

If you know what I've done wrong, and how to correct this fatal flaw, please, point me straight. I can only reward you with Karma Cookies and a blog of response and update.
I acknowledge that CPANTS lists many many passes for this module, so it must be I who is at fault, right?

perl -V

Summary of my perl5 (revision 5 version 13 subversion 4) configuration:
   
  Platform:
    osname=openbsd, osvers=4.7, archname=OpenBSD.i386-openbsd
    uname='openbsd stridor.lan 4.7 generic#558 i386 '
    config_args='-de -Dprefix=/home/kent/perl5/perlbrew/perls/perl-5.13.4 -Dusedevel'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=y, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include',
    optimize='-O2',
    cppflags='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='3.3.5 (propolice)', gccosandvers='openbsd4.7'
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags ='-Wl,-E  -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib
    libs=-lm -lutil -lc
    perllibs=-lm -lutil -lc
    libc=/usr/lib/libc.so.53.1, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
    cccdlflags='-DPIC -fPIC ', lddlflags='-shared -fPIC  -L/usr/local/lib -fstack-protector'


Characteristics of this binary (from libperl): 
  Compile-time options: MYMALLOC PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP
                        PERL_USE_DEVEL USE_LARGE_FILES USE_PERLIO
                        USE_PERL_ATOF
  Built under openbsd
  Compiled at Sep 14 2010 11:31:21
  %ENV:
    PERL5LIB="/home/kent/perl5/perlbrew/perls/current:/home/kent/perl5/perlbrew/perls/current/i386-openbsd"
    PERLDIR="/home/kent/perl5/perlbrew/perls/current"
    PERL_CPANM_OPT="--local-lib=/home/kent/perl5/perlbrew/perls/current"
    PERL_MM_OPT="INSTALL_BASE=/home/kent/perl5/perlbrew/perls/current"
  @INC:
    /home/kent/perl5/perlbrew/perls/current
    /home/kent/perl5/perlbrew/perls/current/i386-openbsd
    /home/kent/perl5/perlbrew/perls/perl-5.13.4/lib/site_perl/5.13.4/OpenBSD.i386-openbsd
    /home/kent/perl5/perlbrew/perls/perl-5.13.4/lib/site_perl/5.13.4
    /home/kent/perl5/perlbrew/perls/perl-5.13.4/lib/5.13.4/OpenBSD.i386-openbsd
    /home/kent/perl5/perlbrew/perls/perl-5.13.4/lib/5.13.4
    .

2010-09-12

Gentoo Protip: Clean orphaned .la files.

If you've been using the "lafilefixer"1 to tweak "broken" .la files, you may have discovered a negative side effect of its use.

Primarily, that lafilefixer breaks the MD5/SHA sums of the various .la files, so when somebody removes that package later, or upgrades to a package with a differently named .la file, it leaves behind this .la cruft.2

The effect of this, is that subsequent builds can die in mysterious ways trying to find stuff, as stupid code tries to use the old and outdated .la files.

The solution is reasonably simple, all you need is a little help from a few good Unix commands.

You'll need 2 basic packages installed:

  1. GNU findutils "xargs" and "find", provided in sys-apps/findutils. You should already have these, because they are after all part of the "system" set.
  2. Gentoo's Portage-Utils'  "qfile", provided in app-portage/portage-utils.

Firstly, we generate a list of all the .la files.

kent@ember$ find -O3 /usr/lib64  -type f  -name "*.la"


We then pipe this list null-delimited to xargs ( for safety ) and ask "qfile" to tell us if they are orphans.

kent@ember$ find -O3 ./  -type f -name "*.la"  -print0 | xargs -0 qfile -o


We can then review this list, make a few "ahh!, so that explains that problem" statements, and then proceed to remove the listed files using our mechanism of choice. xargs + rm is good enough for me.

kent@ember$ find -O3 ./  -type f -name "*.la"  -print0 | xargs -0 qfile -o | xargs rm


And as if by magic, things that no longer wanted to compile resume compiling!.

For me, this cleaned up most of the residual problems I had after the whole libpng12 debacle.

Important: You should read the man pages for xargs and find to make sure you're not just cargoculting bad code. i.e: that -O3 thing can be dangerous.
1:  ( dev-util/lafilefixer )

2: Perhaps portage has a workaround for this, but I'm using Paludis, so don't know, sorry. Complaints to: /dev/null.