mig5.net

I thought I'd document the steps I did to set up a central git repository to which I can push to and pull from for various projects (including the Aegir components), as well as have the repositories sync from the CVS repository on drupal.org regularly.

Before I begin: I'm normally quite verbose and careful with my howtos. However, due to most unwelcome Nagios alerts at 3am on a Sunday morning, I've been awake for about 19 or 20 hours straight. Apologies for any errors with the following instructions, or steps I've missed :)

I also set up git web for HTTP browsing of the repositories, as well as implemented the git:// protocol for anonymous pull requests, alongside using ssh key-based authentication for pushes.

git.mig5.net is running on a Debian 5.0 Xen domU (virtual machine) running on Linode's infrastructure. The following components were apt-get installed to get git and Apache etc working.

apt-get install apache2 git-core git-cvs gitweb

git-core is the base git package in Debian that has *almost* everything one would normally want for standard git development. We also install git-cvs, a separate package, because it provides the git-cvsimport program that is used to incrementally sync CVS repos from drupal.org into a git repo.

Apache will be used for the web server to serve HTTP requests, in this case simply to view the gitweb interface in a browser (though Git does support using HTTP as a protocol for pulling repositories. See the chapter on protocols in the Pro Git book.

Create the path to the git repos
I'll create the paths to the git repos as root, that we will import later.

mkdir -p /drupal/{modules,profiles,themes}
groupadd -g git 
# you might want to the above groupadd if you have multiple developers who will be pushing
# to the repos on this server. You'll want to ensure the git group can write to directories and
# to files. I'm not an expert on git so I'm not 100% sure which files/folders need write access
# in the .git folder. Recklessly, you could do this:
# find /drupal/ -type d | xargs chmod 775
# find /drupal/ -type f | xargs chmod 664
chown -R miguel:git /drupal # (or just chown -R miguel:miguel, no extra git developers)

Import the cvs repositories into new git repositories using git-cvsimport
git-cvsimport is a command available in the git-cvs Debian package that can login to a CVS repository, check out a copy of a module, and convert that checkout into a fully-fledged git repository.

We'll use git-cvsimport regularly to sync changes that are made to the CVS repositories on drupal.org.

First we must login to CVS on drupal.org as the anonymous user.

CVSROOT=:pserver:anonymous@cvs.drupal.org:/cvs/drupal-contrib cvs login

You'll be prompted for a password. Since we're logging in as the anonymous CVS user, the password is 'anonymous'. The password will be hashed and saved to ~/.cvspass. You should only have to do this once.

Now we can use the git cvsimport command. If we were importing a copy of the provision module into /drupal/modules/provision, an example command would look like this:

git cvsimport -v -d :pserver:anonymous@cvs.drupal.org:/cvs/drupal-contrib contributions/modules/provision -C /drupal/modules/provision

It's pretty clear what's happening here: we're logging into CVS on drupal.org, checking out a copy of the provision module from contrib, and converting it into a git repo in the target folder (specified by -C) /drupal/modules/provision. The git database magic gets stored in /drupal/modules/provision/.git, while /drupal/modules/provision itself is a checked out copy of the module.

Automating git cvsimport
git cvsimport will do a full import of a repo, and from then on will perform incremental imports (it will skip previously-imported patchsets). So, we'll want to be doing git cvsimport on a regular basis to drag in new changes. We'll perhaps also want to do this for multiple modules, maybe profiles and themes as well.

For this reason it makes sense to batch this in a simple script, which we can then automate in a cron. Here's a bash script which you can put in /usr/local/bin/cvsimport

#!/bin/bash
 
# An array of modules we want to import into git 
modules=(
  drush
  drush_make
  features
  hosting
  provision
)
 
# An array of profiles we want to import into git 
profiles=(
  hostmaster
)
 
# An array of themes we want to import into git 
themes=(
  eldir
)
 
# Couple of functions for each type of component, which we'll loop through each array of and 
# actually do the cvsimport. Probably could be abstracted out even more to save duplication
# of code
 
function cvsimport_modules {
  for module in ${modules[@]}
  do
    git cvsimport -v -d :pserver:anonymous@cvs.drupal.org:/cvs/drupal-contrib contributions/modules/$module -C /drupal/module
s/$module
  done
}
 
function cvsimport_profiles {
  for profile in ${profiles[@]}
  do
    git cvsimport -v -d :pserver:anonymous@cvs.drupal.org:/cvs/drupal-contrib contributions/profiles/$profile -C /drupal/profiles/$profile
  done
}
 
function cvsimport_themes {
  for theme in ${themes[@]}
  do
    git cvsimport -v -d :pserver:anonymous@cvs.drupal.org:/cvs/drupal-contrib contributions/themes/$theme -C /drupal/themes/$theme
  done
}
 
# Run the actual functions
 
cvsimport_modules   
cvsimport_profiles  
cvsimport_themes

As you can see, we specify an array of components in groups of modules, profiles or themes, and for each case we run a cvsimport on it. Now we can simply write a cron that will run this script every half hour or so:

/etc/cron.d/cvsimport

*/30 * * * *    miguel  /usr/local/bin/cvsimport > /dev/null

Every half hour the repos will be scanned and imported incrementally into /drupal/$component-type/$module. Your git repos are up and ready to be used.

Adding new repos to sync over from CVS on drupal.org is a trivial matter of adding that component name to the relevant array in the script, and on next cron run (or run it manually), they'll be dragged across. Easy!

Create a browser-based frontend for viewing the repos, using git-web
I used git-web to hook into Apache so I could browse the git repos via HTTP in my web browser.

On Debian, the gitweb package installs a config file into /etc/gitweb.conf. I edited it to look like this (only the $projectroot entry was changed):

# path to git projects (<project>.git)
$projectroot = "/drupal";
 
# directory to use for temp files
$git_temp = "/tmp";
 
# target of the home link on top of all pages
#$home_link = $my_uri || "/";
 
# html text to include at home page
$home_text = "indextext.html";
 
# file with project list; by default, simply scan the projectroot dir.
$projects_list = $projectroot;
 
# stylesheet to use
$stylesheet = "/gitweb.css";
 
# logo to use
$logo = "/git-logo.png";
 
# the 'favicon'
$favicon = "/git-favicon.png";

Since this server was not going to host any other vhosts other than git.mig5.net, I just fiddled with the apache default host in /etc/apache2/sites-available/default. It looks like this:

<VirtualHost *:80>
        ServerAdmin webmaster@mig5.net
        DocumentRoot /var/www
        <Directory />
                Options Indexes FollowSymLinks ExecCGI
                AllowOverride None
                DirectoryIndex /cgi-bin/gitweb.cgi
        </Directory>
        ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
        <Directory "/usr/lib/cgi-bin">
                AllowOverride None
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                Order allow,deny
                Allow from all
        </Directory>
        ErrorLog /var/log/apache2/error.log
        CustomLog /var/log/apache2/access.log combined
</VirtualHost>

After an Apache restart, things are working at http://git.mig5.net, but no styling or images are working. I just copied the files from /usr/share/gitweb/* to /var/www/

cp /usr/share/gitweb/* /var/www/

You'll note that by default, the repository descriptions in the web interface are by default something like this:

Unnamed repository; edit this file to name it for gitweb.

To change this, edit the 'description' file in each repository. For instance, to change this description for the Hosting module, I edit /drupal/modules/hosting/.git/description and simply replaced it with

Hosting module - frontend of the Aegir hosting system. master branch synced with Drupal.org CVS every 30 minutes.

No restart is required for this change to take effect.

Start the git-daemon to allow git:// protocol for anonymous cloning/fetching/pulling etc
So I push to my git repositories using standard SSH protocol (using key-based authentication - more on this later). However, I want you lot to be able to clone my repos and checkout my working branches etc anonymously, without any further action on my part!
The answer it to implement the git:// protocol using the git-daemon program, which opens up TCP port 9418 by default and listens for inbound 'upload-pack' requests. This allows anonymous users to clone my repos likeso:

git clone git://git.mig5.net/drupal/profiles/hostmaster

It was at this point I realised I probably should have put my git repos somewhere other than in the / directory, since git-daemon takes a --base-path argument and I was nervous about specifying --base-path=/ while wanting to preserve consistency with git clones over SSH per 'git clone git.mig5.net:/drupal/modules/hosting' etc..

So I created a symlink in /opt/git that pointed to /drupal:

mkdir -p /opt/git
cd /opt/git
ln -s /drupal/ 

The git-daemon process can then be initiated likeso:

git-daemon --verbose --export-all --base-path=/opt/git

This exports every .git repo in /drupal to the git:// protocol, allowing them to be cloned. However at this point the process is running in the foreground. To daemonize the git-daemon process, I wrote a little /etc/init.d/git-daemon script likeso:

#!/bin/bash
 
# Start the git daemon so that people can pull
base_path=/opt/git
 
git-daemon --detach --syslog --verbose --export-all --base-path=$base_path

(I know, it ought to really have start|stop|restart case statements, but as I said earlier, I'm very tired :)

This detaches the process to the background and also logs verbose output to /var/log/syslog.

I added this script to the startup processes with the standard update-rc.d (obviously made this script executable first)

update-rc.d git-daemon defaults

Pushing to the repos over SSH
Pushing to your central repo from other machines is as simple as adding the central server's repo as a remote SSH repo from your local cloned copy:

git remote add mig5 git.mig5.net:/drupal/modules/provision

Generate an ssh key pair on your local machine if you haven't one already:

ssh-keygen -t rsa

Then copy the ~/.ssh/id_rsa.pub content to ~/.ssh/authorized_keys of the corresponding user on the git central server.

Once your ssh key stuff is in place, you can simply push to the remote after commiting your changes locally:

git push mig5 master

(or pushing a local branch to a remote branch:)

git push mig5 my_branch

If this is a new branch, this branch will appear in the 'heads' section in gitweb. Here's an example of the branches of the Hosting module on git.mig5.net

That's really it for getting your git central repo going, both for anonymous cloning, browser-based viewing, and ssh pushing! I didn't go through general git commands and practices. I really highly recommend having a read of the Pro Git book, published by Apress but available free in its entirety online at that address.

I also didn't go through pushing git commits back to CVS directly (for CVS, I've a habit of simply doing a git diff on my working branch against the master, and patching the CVS checkout before committing back to CVS). There is a lot of Drupal/Git/CVS documentation floating about elsewhere too, for example here, as well as anarcat's git workflow docs which touches on this.

Also, Damien Tournoud mentioned that he's implemented an entire git mirror of Drupal core + contrib on git.drupalfr.org.