Making a Minting Profile as a CPANized Dist.

... or one more reason why Dist::Zilla is awesome.

Minting With Dzil

Minting is a term we use for "Creating a new distribution from a template of sorts". Since Dist::Zilla 2.101230 ( 2010-05-03 23:42:27 America/New_York ), there has been a command for Dist::Zilla to facilitate setting up a new distribution.

$ dzil new Acme-An-Example
# [DZ] making target dir /tmp/Acme-An-Example
# [DZ] writing files to /tmp/Acme-An-Example
# [DZ] dist minted in ./Acme-An-Example
$ find Acme-An-Example/
# Acme-An-Example/
# Acme-An-Example/dist.ini
# Acme-An-Example/lib
# Acme-An-Example/lib/Acme
# Acme-An-Example/lib/Acme/An
# Acme-An-Example/lib/Acme/An/Example.pm
$ cat Acme-An-Example/dist.ini
# name    = Acme-An-Example
# author  = Kent Fredric 
# license = Perl_5
# copyright_holder = Kent Fredric 
# copyright_year   = 2011
# 
# version = 0.001
# 
# [@Basic]
$ cat Acme-An-Example/lib/Acme/An/Example.pm
# use strict;
# use warnings;
# package Acme::An::Example;
# 
# 1;

And while this is a good starting point, once you've gotten into Dist::Zilla, you'll probably find it somewhat lacking and find yourself doing things over and over again.

For this, we have "Minting Profiles".

Basic Minting Profiles

.

Minting Profiles have been in Dist::Zilla since 4.101780 ( 2010-06-27 14:30:55 America/New_York ).

The first thing you'll tend to do, is make a minting profile in ~/.dzil

dzil.org's minting-profile tutorial adequately covers the innards of how this works, but summarised, its lots like dist.ini. You have a profile.ini in a directory in ~/.dzil/profiles/$profilename/ and it will control how your dist is created, and you can then throw together new dists as simply as:

$ dzil new -p $profilename Acme-An-Example

In the absence of -p $profilename, if there is a ~/.dzil/profiles/default/, that will be used instead.

This is pretty convenient.

Me, I had a very simple default profile for a while that did most of what I needed to do:

; ~/.dzil/profiles/default/profile.ini
[Author::KENTNL::DistINI]

That behaves more-or-less identical to how normal dzil new operates, but it generates a custom dist.ini from my custom Author::KENTNL::DistINI plugin. This is based on DistINI just tuned for how I like it. Its more or less a template ;).

; Generated by Dist::Zilla::Plugin::Author::KENTNL::DistINI version 0.01023312 at Fri Jan  7 20:05:56 2011
name             = Acme-An-Example
author           = Kent Fredric 
license          = Perl_5
copyright_holder = Kent Fredric 

; Uncomment this to bootstrap via self 
; [Bootstrap::lib]

[@KENTNL]
version_major     = 0
version_minor     = 1
; the following data denotes when this minor was minted
version_rel_year  = 2011
version_rel_month = 1
version_rel_day   = 7
version_rel_hour  = 20
version_rel_time_zone = Pacific/Auckland
twitter_hash_tags = #perl #cpan

[Prereqs]

CPAN Centralization

However, many people like to use CPAN as their file storage mechanism, and centralise all their perly bits they might need in different locations so they can just get their current favourite configuration with a handy :

cpanm --interactive --verbose Dist::Zilla::MintingProfile::Author::KENTNL 

A Word on Name-spaces

Before we go further, I think it important to plead my rationale behind the "Author::" prefix I'm using now, as I feel its an important concern.

As we have seen with Dist::Zilla there have been a slew of PluginBundles with CPANID's in their name, to the point that there is a copious amount of name-space pollution in the PluginBundle name-space, and more Author bundles than task-bundles, which was really what the name-space was designed for, and I'll freely admit, I am guilty of this crime as well, but I'm petitioning you to help reduce this annoyance in future modules.

From a CPAN testers perspective, the annoyance of lots of CPANID-dists is similar to the annoyance of the whole DPCHRIST:: subspace, and that if this pattern continues, it will mean for the testers who do not wish to test everyones personal modules, that they will have to work hard to avoid this. If DPCHRIST:: had used something like Author::DPCHRIST:: instead, I doubt so many people would be horrified by it, because you can just have a policy/rule that excludes ^Author::, and everyone else who goes that way can be quietly ignored.

Then we could probably rationally add that same restriction to the irc announce bots, the "recent modules" list and soforth, and possibly even apply special indexing restrictions or something so people wouldn't even have to know those modules exist on cpan!

So, for the sake of cleanliness, semantics, and general global sanity, I ask you to join me with my Author:: naming policy to voluntarily segregate modules that are most likely of only personal use from those that have more general application.

Dist::Zilla::Plugin::Foo                 # [Foo]                 dist-zilla plugins for general use
Dist::Zilla::Plugin::Author::KENTNL::Foo # [Author::KENTNL::Foo] foo that only KENTNL will probably have use for
Dist::Zilla::PluginBundle::Classic       # [@Classic]            A bundle that can have practical use by many
Dist::Zilla::PluginBundle::Author::KENTNL #[@Author::KENTNL]     KENTNL's primary plugin bundle
Dist::Zilla::MintingProfile::Default     # A minting profile that is used by all
Dist::Zilla::MintingProfile::Author::KENTNL # A minting profile that only KENTNL will find of use.

Making Dist::Zilla::MintingProfile::Author::YOURID

First, we'll start with the above default empty profile with a basic dist.ini

$ dzil new Dist-Zilla-MintingProfile-Author-YOURID

And then the guts of that Minting Profile is pretty basic.

use strict;
use warnings;

package Dist::Zilla::MintingProfile::Author::YOURID;

# ABSTRACT: YOURID's Minting Profile

use Moose;
use namespace::autoclean;
with 'Dist::Zilla::Role::MintingProfile::ShareDir';


__PACKAGE__->meta->make_immutable;
no Moose;
1;

This is a deceptively simple piece of magic that took me a little while to fully grok.

Next, we make a 'profile' directory in our dist.

$ mkdir -p Dist-Zilla-MintingProfile-Author-YOURID/share/profiles/

And in that, we can put one or many profiles. For simplicity, we'll just start off with a 'default' profile.

$ mkdir -p Dist-Zilla-MintingProfile-Author-YOURID/share/profiles/default/
$ pushd Dist-Zilla-MintingProfile-Author-YOURID/share/profiles/default/

Now, you could just stick the same profile.ini you had above in ~/.dzil there, and you're mostly done with the hard part. However, I'll go a little more into detail so you get an idea of what use it is.

Make a 'skel' dir

Make a 'skel' dir holding a selection of templated-files you want in all new dists

Here is the most notable ones in mine

$ mkdir skel
# stuff directory with files
$ find skel/
# skel/
# skel/.gitignore
# skel/.perltidyrc
# skel/Changes
# skel/perlcritic.rc
# skel/weaver.ini
$ cat skel/Changes
# Release history for {{ $dist->name }}
# 
# {{ '{{$NEXT}}' }}
#
#        First version, released on an unsuspecting world.
#
$ cat skel/.gitignore
# .build
# {{ $dist->name }}-*

{{ $dist->name }} is expanded during 'new' to the name of the intended dist, and {{ '{{$NEXT}}' }} is just a nasty hack so that the literal string {{$NEXT}} turns up in the changelog file so the Changelog plugin keeps working.

For more in-depth understanding, please read The dzil minting profile tutorial

Adding support for the skel directory

You now need to tell your profile.ini about that skel directory if you want it to mean anything.

[GatherDir::Template]
root = skel
include_dotfiles = 1 ; I want the dot files! 

./profile.ini    # this file ---> indicates inclusion of
./skel                           #      /
./skel/.gitignore          <-----------/
./skel/.perltidyrc         <----------/
./skel/Changes             <---------/
./skel/perlcritic.rc       <--------/
./skel/weaver.ini          <-------/

Other misc profile additions

Now most people will probably just put in

./skel/dist.ini

With some template variables expanded, but I myself prefer the dedicated module approach for dist.ini, so I add that to profile.ini

[GatherDir::Template]
root = skel
include_dotfiles = 1 ; I want the dot files! 

[Author::KENTNL::DistINI]

And for good measure, I want every new dist to automatically have git set up with it, so I add that plugin too.

[GatherDir::Template]
root = skel
include_dotfiles = 1 ; I want the dot files! 

[Author::KENTNL::DistINI]
[Git::Init]

Gluing it together

Now if you've been following me, you'll have a directory that looks a lot like this:

dist.ini
lib/
lib/Dist
lib/Dist/Zilla
lib/Dist/Zilla/MintingProfile
lib/Dist/Zilla/MintingProfile/Author
lib/Dist/Zilla/MintingProfile/Author/YOURID.pm
share/
share/profiles
share/profiles/default
share/profiles/default/profile.ini
share/profiles/default/skel
share/profiles/default/skel/.gitignore
share/profiles/default/skel/.perltidyrc
share/profiles/default/skel/Changes
share/profiles/default/skel/perlcritic.rc
share/profiles/default/skel/weaver.ini

And you're probably wondering how the 'share' directory and its contents will be seen by 'YOURID.pm' after installation.

The magic is this stanza in your dist.ini

[ModuleShareDirs]
Dist::Zilla::MintingProfile::Author::YOURID = share/profiles 

Yeah, at first that baffled me too.

Into the guts of File::ShareDir

Dist::Zilla utilizes sharedir in 2 places, 1. In the module that does the minting, and 2. Using some instructions injected in your install tool.

During Install

Lets pretend there is a directory, we'll call it $x, which is known as the "base directory" for all Things File::ShareDir::Install installs, which is also known to to File::ShareDir.

For example:

/usr/lib64/perl5/vendor_perl/5.12.2/auto/share/    # $x 
/usr/lib64/perl5/vendor_perl/5.12.2/               # module install dir.

The ModuleShareDirs plugin tells File::ShareDir::Install that the directory 'share/profiles' should be installed in a path associated with the module, i.e.:

$x/module/Dist-Zilla-MintingProfile-Author-YOURID/ <= share/profiles/ 

So, during install, the files in share/profiles/* will be copied to that directory.

.../auto/share/module/Dist-Zilla-MintingProfile-Author-YOURID/default/skel/perlcritic.rc
.../auto/share/module/Dist-Zilla-MintingProfile-Author-YOURID/default/skel/weaver.ini
.../auto/share/module/Dist-Zilla-MintingProfile-Author-YOURID/default/skel/.perltidyrc
.../auto/share/module/Dist-Zilla-MintingProfile-Author-YOURID/default/skel/.gitignore
.../auto/share/module/Dist-Zilla-MintingProfile-Author-YOURID/default/skel/Changes
.../auto/share/module/Dist-Zilla-MintingProfile-Author-YOURID/default/profile.ini
.../Dist/Zilla/MintingProfile/Author/YOURID.pm

And that is not all that scary =)

How your profile uses that

Having seen how it installs to the file-system, how the plug-in works should be a bit of a no brainer.

• dzil new -P Author::YOURID -p default Some-Module-Name

• Dist::Zilla, sees -P, and loads the respective profile from @INC, Dist::Zilla::MintingProfile::Author::YOURID
• Your minting profile uses that sharedir role, and asks File::ShareDir for the directory associated with 'Dist-Zilla-MintingProfile-Author-YOURID', which of course returns that '.../auto/share/module/Dist-Zilla-MintingProfile-Author-YOURID/' we installed above.
• The Role then determines which sub-profile to use ( default, as specified by -p ) and then returns that path ( now .../auto/share/module/Dist-Zilla-MintingProfile-Author-YOURID/default/ )
• .../auto/share/module/Dist-Zilla-MintingProfile-Author-YOURID/default/profile.ini is then read and a new distribution is constructed using the instructions therein
• And presto, you have a newly minted dist =):

  $ cd Some-Module-Name;
  $ find
  # ./.git 
  # ... a bunch of .git files ...
  # ./dist.ini
  # ./weaver.ini
  # ./perlcritic.rc
  # ./Changes
  # ./.perltidyrc
  # ./.gitignore
  # ./lib
  # ./lib/Some
  # ./lib/Some/Module
  # ./lib/Some/Module/Name.pm
  

Hopefully thats enough to get you started, but...

There is one warning you must uptake with the use of file sharedir.

ONCE YOU INSTALL A FILE IN A SHAREDIR FROM A CPAN DIST, IT IS THERE FOR GOOD

At least, it will be there till somebody manually deletes it, at least in the current implementation of CPAN.

This means if you decide to put a file in skel/, and then later decide you don't want that file in skel/, you're going to have a problem when something decides to "add all of skel/ to your new dist".

For vendor-wrapped/packaged stuff ( ie: Redhat, Debian, Gentoo ) you're fine, those systems have package management that guarantee atomicity somewhat, but the CPAN toolchain ( at least, in my experience ) provides very little guarantee in this regard. This may change in the future, but that's how it seems to be now. But now I've warned you, you'll hopefully not bump into that =).

Dist-Zilla-Plugin-Git-CommitBuild

I've contemplated this for a while. I might get a round tuit, and do this myself, so this blog entry is here to jog my brain, jot down ideas, possibly collect info.

If you look at the repositories behind any of my CPAN dists ( well, most of them ), you'll see I maintain both release and source branches for the entire history ( http://github.com/kentfredric/ELF-Extract-Sections/network ), and more recently, maintaining a sort of "pre-release/release" sub-system, where stuff I build just for testing/preview purposes may have a life on their own branch, sort of like release candidates. 

This is essentially to provide a branch that is containing a full copy of all the generated code, as posted on CPAN, as opposed to the source that it is generated from, for posterity reasons mostly, and so I can deprecate versions on CPAN for incompatibility reasons one day , and people won't be left in the lurch to get an identical copy of it somewhere, as it will always be in the git history, just grab the right tag and you're set.

They could always use the backpan, but that has 2 caveats in my experience.

1.  No diff mechanism. This feature is very important to people who do release maintenance for distributions, as  its the only good way to conclusively see what exactly changed between 2 consecutive versions, in order to update their internal dependency data that controls the shipping of the built copies.
   
   For this reason also, I loathe every time somebody deletes an older copy of their dist when its not been outdated for < 3 months, because it can take that long to notice that the shipped copy is outdated and for somebody to request a version bump. Not being able to use CPAN's diff feature makes this task much more challenging. ( At least for me, for that is how I do my work-flow, and I kind-of help out lots with gentoo's perl-experimental overlay ).
   
   
2. Sometimes, versions live too short a time to be backed up on backpan. This is very problematic, for the above reason, and for the reason is you have no historical record of what happened outside the Changes, and the original commit history.  You could probably argue there's no reason to ever want these version that never made it to backpan, and you'd probably be right.
   

So to remedy this problem, here is what I do.

1. Commit, and tag the exact source tree that was used to generate the released code in the notation %v-source. This theoretically guarantees that anyone can check out that exact same release, run "dzil release", and produce more or less the exact same output, with the only difference possibly being the version numbers emitted if you're using an [AutoVersion] or [AutoVersion::Relative] plugin.
   
   
   
   Here is the code snippet I use to do this that uses Jerome Quelin's [Git] plugin suite.
   

   [Git::Check]
   
   filename = Changes
   
   
   
   [NextRelease]
   
   
   
   [Git::Tag]
   
   filename = Changes
   
   tag_format = %v-source
   
   
   
   [Git::Commit]
   
   

   
   

   This Order is important. In the Build phase, [NextRelease] formats the Changes template into an exportable form, and puts the datestamp in it.
   
   
   
   In the pre-release phase, [Git::Check] makes sure theres nothing in the tree that isn't committed.
   
   
   
   [UploadToCpan]uploads the dist to CPAN, and the post-release phase kicks in.
   
   
   [NextRelease]  then kicks in again, and reformats the Changes so it resembles the previously released Changes except with that {{$NEXT}} stuff in it ready for hacking on. 
   
   
   
   [Git::Tag]  tags the last commit ( that is, not the current tree with the modified Changes, that's not committed yet, but the commit that it was at still when we released ) with %v-source, and then [Git::Commit] commits the updated Changes as a new commit ( with a copy of the first segment of the Changes file as its commit message )
   
   
   
2. Have a separate commit history just for releases to be copied into.
   

   git symbolic-ref HEAD refs/heads/releases
   

   The first commit of this is built, generally from the first releases files. At present, I do this first release as so:
   

   rsync -avp Some-Dist-Name-0.010101/ ./
   

   then weed out all the files I'm pretty sure weren't in the generated tree by hand. ( I had a some code that did it all with rsync, and had an ignore list so that the --delete-after argument didn't accidentally erase all of .git, which would be very sad, but I accidentally deleted it :[  )
   
   
   
   This tree now represents an exact copy of the generated code, and it is committed as follows:
   

   git commit -m "Build of 0deadbeef0, version 0.010101 on cpan"
   

   or similar, to assure that every commit on the release branch, is a direct derivative of another commit on the source branch, and there's an intrinsic link between them.  ( I avoided having a direct link, because that gives cleaner histories ).
   
   
   
3. That commit is tagged as the released version, ( ie: 0.010101 )

Now all this is wonderfully Tedious. At present, the best I have a script that makes the "commit and tag" phase on the release branch reasonably painless, but what I want to do, is have a nice way, to automate all of the above, every single bit of it, with a plugin.

Here is some proposed syntax.

[Git::CommitBuild / prerelease ]

branch = prereleases

autocreate = 1

phase = build



[Git::CommitBuild / release]

branch = releases

autocreate = 1

phase = after_release

 Why this notation? well, I guess it just seems the right amount of flexible to me.
the text after the  / is totally optional, and its just a way to let dzil differentiate between copies of the same plugin.  

branch is to tell it what git branch to work with. I figgured I could just use the name part after the  /, but it seemed nasty to me ( spaces for instance ). At very best, it could default to that value if branch = is not specified.

autocreate = 1 would magick the branch out of fat air the first time you tried to commit to it and it wasn't there. This would be off by default, as it could be annoying to you if you'd already created another branch with a different name for that purpose, and typoed and it created another branch. This way it fails instead of annoying you.

phase = build is sadly the most scary bit I'm trying to eliminate the stink of. Essentially, I have one plugin that does only one thing, but there are 2 different times I may want to run it at. ( And there are possibly more places people might want to say "stop the build, store this somewhere, then continue" ). 

In the above scenario, 'prerelease' I envisage as only getting run when I call "dzil build" explicitly. NOT 'dzil test' and NOT 'dzil release', only 'dzil build'.

Also, Ideally, the whole commit phase should be done, magically, entirely in memory, with some magical git magic, to eliminate the whole "write it out to the filesystem before creating the actual commit data" part of the equation, so that nowhere anywhere does there transpire something like

git checkout releases
git commit stuff
git checkout master

which causes anarchy in the event anything else happened to be using the file-system.

Thoughts/Suggestions anyone?