During testing, I discovered that the odd-ball task failed to run, and found the following error in the system log:

sudo: sorry, you must have a tty to run sudo

I traced this error to a line trying to invoke a perl command as a user called dynamic:

sudo -u dynamic /usr/bin/perl run-periodic-tasks --load 5 --randomly

A simple Google search turned up an obvious solution to the error: use visudo to disable sudo’s tty requirement, allowing sudo to be invoked from any shell lacking a tty (including cron). This would have solved my problem, but it just felt wrong, dirty, and most troublingly insecure.

One reason why sudo ships with the requiretty option enabled by default is, among other reasons, to prevent remote users from exposing the root password over SSH. Disabling this security precaution for a simple maintenance task already running as root seemed totally unnecessary, not to mention irresponsible. Moreover, client‘s script didn’t even need a tty.

Thankfully, there’s a better way: use su --session-command and send the whole job to the background.

su --session-command="/usr/bin/perl run-periodic-tasks --load 5 --randomly" dynamic &

This line launches a new, non-login shell (typically bash) as the other user in a separate, background process and runs the command you passed using the shell’s -c option. Sending the command to the background (using &) continues execution of the rest of the cron job.

A process listing would look like this:

root     28109     1  0 17:10 ?        00:00:00 su --session-command=/usr/bin/perl run-periodic-tasks --load 5 --randomly dynamic
dynamic  28110 28109  0 17:10 ?        00:00:00 bash -c /usr/bin/perl run-periodic-tasks --load 5 --randomly

Note the parent process (PID 28109) is owned by root but the actual perl process (PID 28110) is being run as dynamic.

This in-script solution that replaces sudo -u user cmd with su --session-command=cmd user seems much better than relying on a change in sudo‘s default (and more secure) configuration to me.

Note that with the set up described in the above-linked previous post, you can only pull over HTTP. This is usually what you want. If you want to be able to push over HTTP as well, git must be compiled with the USE_CURL_MULTI flag.

This is, as it turns out, because git seems to use curl for its HTTP operations, which also obviously means you must have curl installed on your workstation if you don’t already and it also implies that it’s curl, not git which you need to configure. In other words, accessing a git repository that is behind HTTP authentication is exactly the same as accessing one without it, and so is publishing a git repository to an HTTP server. The rest of this short tutorial assumes you have published your repository at http://example.com/git/public-repo.git and are using the Apache web server.

Step 1: Create an HTTP Basic Authentication username and password file

First, you’ll need to create a file that lists the usernames who are permitted to access your repository over HTTP Basic authentication. This is easily accomplished with the htpasswd utility (or your host’s custom web UI, if one is provided). Let’s create a file called .git-htpasswd to store these usernames and passwords.

From your shell, run the following command:

htpasswd -c /path/to/DOCUMENT_ROOT/.git-htpasswd username

where /path/to/DOCUMENT_ROOT is the full path to the root directory of your web site and username is the username you want to add. If you want to add subsequent users to this file, run the same command again without the -c, like this:

htpasswd /path/to/DOCUMENT_ROOT/.git-htpasswd another_username

You’ll then be prompted to enter a password, and then prompted again to verify that you’ve typed it correctly.

Step 2: Configure HTTP Basic Authentication on Apache

Next, configure standard HTTP Basic Authentication on Apache. In most shared hosting environments, you’ll be allowed to configure per-directory passwords using .htaccess files. Some hosts provide web UI interfaces for creating “protected folders,” which is basically the same thing. Make certain that the kind of protection you select is “Basic,” because curl will require that.

To do that, create a new file named .htaccess in your DOCUMENT_ROOT/git directory if one does not already exist with the following contents:

AuthType Basic
AuthName "Git"
AuthUserFile /path/to/DOCUMENT_ROOT/.git-htpasswd
Require valid-user

This tells Apache to look for usernames and passwords in the file named .git-htpasswd we created in step 1.

If everything is set up correctly, you should now be able to access http://example.com/git/public-repo.git in your Web browser and you should be presented with a login dialogue box.

Step 3: Configure curl on your (client) workstation computer

Next, configure your local curl client. git-pull will call curl with its --netrc-optional switch for HTTP operations. This means curl will look for a file named .netrc in your home directory and will read authentication configurations from that file. The format of this file is incredibly simple:

machine yourserver.example.com
username your_username
password your_password

To check if this is working correctly, run curl yourself to access the current HEAD of the public repository and see if you get the expected result:

curl --netrc --location -v http://example.com/git/public-repo.git/HEAD | grep 'ref: refs/heads'

If you see a line of output then you know this is working, otherwise you should double check your work.

Step 4: There is no step four

You’re done. With this configuration, you can git-pull as you normally would, and git will automatically use your .netrc file to enable curl‘s HTTP authentication schemes.

cd ~/                          # change to home directory
test -d ~/src || mkdir ~/src   # if there isn't already a ~/src directory, create it
cd ~/src                       # then change to that directory
curl -O http://www.kernel.org/pub/software/scm/git/git-1.5.6.4.tar.gz # download
tar -xvzf git-1.5.6.4.tar.gz   # and extract the git source code
cd git-1.5.6.4                 # change to the source code directory
./configure --prefix=$HOME     # configure build to install into $HOME
make                           # do the build
make install                   # move the built binaries to the right places
echo "export PATH=\$PATH:$HOME/bin" >> ~/.bashrc # make sure non-interactive shells can find git

Of special note is the last line, which sets up the necessary $PATH specifically for non-interactive bash shells for use with git-push or git-pull. With out that, you’ll run into the infamous “bash: git-receive-pack: command not found” error.

Also, of course, lines 4 through 6 are referring to version 1.5.6.4 of the git tarballs, so you may want to change these to point to whatever is now the most recent version.

In these cases, it makes sense to be smart about when I initiate these downloads. Being something of a UNIX-head myself, I wanted to use the age-old at command to download a Linux ISO during off-peak hours, which my ISP says starts at 2 AM. Much to my chagrin, I found that at doesn’t work by default on Mac OS X and, worse, the Leopard man page leads to a dead end (though it didn’t back in Tiger…).

Turns out that the system daemon that is responsible for checking up on at jobs has been wrapped with a launchd job. This makes enabling at on your system really easy:

sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.atrun.plist

Once you’ve done this, you can now use at as you normally have done. For instance, I could now schedule my downloads to happen during the off-peak hours:

Perseus:Fedora maymay$ at 2:15am tomorrow # now press return
curl -LO http://download.fedoraproject.org/pub/fedora/linux/releases/9/Fedora/x86_64/iso/Fedora-9-x86_64-DVD.iso
# now press CTRL-D.
job 1 at Tue Jul 15 02:15:00 2008
Perseus:Fedora maymay$ atq
1	Tue Jul 15 02:15:00 2008

This is also incredibly handy for scheduling just about any resource-intensive task that you don’t have to do right now. To take it one step further, you can even let the computer itself choose when to run these resource-heavy tasks by using the batch command, which will execute commands much like at but will check the system load average instead of the system clock to determine if it should start the job.

Note that with the com.apple.atrun job loaded /usr/libexec/atrun is started every 30 seconds (unless you change the StartInterval key in the plist file). Since the atrun command checks a file on disk (that it places in the /usr/lib/cron/jobs directory) to see if there is any work to do, this will probably prevent your disks from ever sleeping, which could be a major concern for battery life on portables. Also, obviously, your computer needs to be turned on and awake for the job to actually launch.

For more information, check out the result of typing man at and man launchctl at a Terminal prompt. There’s also a really good Google Tech Talk about Launchd that will teach you a lot more about job scheduling on Mac OS X.

However, if the services you typically access via Bonjour use static IP addresses, then there is one age-old networking technique you can use to simulate Bonjour-style naming conventions without actually using Bonjour. This, of course, is the /etc/hosts file.

The /etc/hosts is a simple, static, text-based mapping of computer names to IP addresses. It does exactly what Bonjour does except it doesn’t keep itself up to date when things change. Of course, if you’re using static IPs for the services you want access to, you can pretty safely assume that things aren’t going to be changing frequently anyway. Long-time sysadmins will laugh at this, but I say let them laugh. This is remarkably useful and very easy to implement.

Let’s assume I’m running a personal web server on my home network, and I can access my home network via a VPN. On my home network, my web server’s IP address is, say, 192.168.2.100, and I usually access it as http://server.local/. All I need to do is open a Terminal prompt and run the following commands as an administrative user:

sudo echo "192.168.2.100	server.local" >> /etc/hosts

That’s it. What this does is hard-wire the name server.local so that it always resolves to the IP address 192.168.2.100. Now, anytime anything on my computer tries to access server.local, it’ll always access 192.168.2.100 directly instead of ever needing to make an mDNS query on the network. The net effect is that we can trick our computer into thinking that Bonjour is working, even when it’s not—such as over a VPN connection.

Note that in default cases, hard-wiring an IP address like this completely prevents your computer from ever asking other computers (such as DNS servers) what the current IP address for this name is. That means if the IP address of the remote server changes, you won’t be notified, and things will just not work. So be mindful that you’ve made this change, and revert it as a first step in troubleshooting procedures.

By the way, Windows users can do the very same thing simply by editing their etc/hosts. They can find this file at C:\WINDOWS\system32\drivers\etc\hosts and can edit it with Notepad. They will also need to install Bonjour for Windows to get Bonjour working in the first place, of course.

for i in *; do
    v=`grep 'version = "' "$i/current/$i/"*.info |
      cut -d ':' -f 2 |
        sed -e 's/^version = "/v/' -e 's/"$//'`
    svn cp "$i/current" "$i/$v"
done;

It’s a bit buggy for some modules that have multiple .info files, but I’m sure a few more pipeline stages can fix that. (Which, because I’m done with this at the moment, I will leave as an exercise to the reader.)

Chalk this one up as another testament to the power of shell scripting and how it can help every developer get their job done faster.

With that caveat out of the way, here’s how I managed to move all the pages in an Apple WikiServer group wiki and blog from one group to another.

Dear God, please have a backup!

Okay, step 1 is mundane, but seriously, please of please have a backup of your data before you do any of these things. To make a back up of your entire WikiServer’s data store, simply:

sudo tar -cvzf backup-file-name.tgz /Library/Collaboration

If anything goes wrong, you can restore from your backup just as simply:

sudo tar -C /Library/Collaboration -xvzf backup-file-name.tgz

Okay, with that out of the way, next make sure absolutely nobody is using any of the Group services for the group you are going to perform the move from, or to. People can work on other group’s wikis, it’s just the ones you’ll be touching you want people to avoid. You can enforce this with some Apache redirects, which is left as an exercise to the reader.

Step 1: Rename or create a new Group in Workgroup Manager

First, you need to either rename or create a new group in Workgroup Manager. If you make a new group, be sure to enable all the services that the old group used.

If you’re simply renaming a group, then you might not even have to go through this trouble. You merely need to change the group’s “Name” (as opposed to its “Short Name”) and then stop and start the Web Service from Server Admin to see the group’s name change. However, if you also want to change the group’s “Short Name” (i.e., the group’s POSIX group account symbolic name), then you will need to perform these steps.

Once you have the new group ready to go in Workgroup Manager and you have stopped and started your Web Service in Server Admin, continue to the next step.

Step 2: rsync all your files from the old group to the new group

This is simple. Just run:

sudo rsync -avzE --progress /Library/Collaboration/Groups/old-group/ /Library/Collaboration/Groups/new-group/

Note that the trailing slashes on the directory names in this command are quite important, as they tell rsync to take the contents of the first directory and place those items as the contents of the second directory. Without the trailing slashes, rsync will make extraneous directories for you, which Apple WikiServer won’t understand. See man rsync for more information.

Step 3: Update the plists for all your WikiServer pages in the new group

Now, WikiServer has all the content of your old group but all of the internal references are wrong, since they still point to the old group. What you need to do is rewrite all those references so that they point to the new group address. The group references are stored in property list files. See my older blog entries for details about the filesystem structure of Apple’s WikiServer data storage layout.

To do this is a relatively simple procedure. As root, do the following, and do it carefully because as root you can easily mess up (and there is no undo button on the command line!):

sudo su -
cd /Library/Collaboration/Groups/new-group
grep -ri "old-group" * | grep '^[^B]' | cut -d ':' -f 1 | grep 'plist$' | sort | uniq > /tmp/list_of_plist_files_to_edit
for i in `cat /tmp/list_of_plist_files_to_edit`; do sed -e 's/<string>groups\/old-group/<string>groups\/new-group/' $i > $i.new; mv "$i.new" "$i"; chown teamsserver:teamsserver "$i"; chmod o-rwx "$i" done;

In English, this means:

1. Become root.
2. Change to the /Library/Collaboration/Groups/new-group directory
3. Find all plist files that have the the string old-group in them, sort them, and write this list of files to the /tmp/list_of_plist_files_to_edit file.
4. For each of the files listed in the /tmp/list_of_plist_files_to_edit, find the text string <string>groups/old-group and replace that text with <string>groups/new-group, and save this change as the name of the file with .new appended. Finally, replace the original file with the file we modified, and give them the appropriate permissions.

Step 4: Create an Apache redirect to make sure no HTML links are broken

Okay, at this point your new group is up and running and it should be working. However, if you had any links at all in any of your group’s pages, they are now all broken because they still point to the old group. Rather than going through the HTML itself and cleaning this up right now (because that’s very error-prone indeed, even with automated tools), it’s much easier to just tell Apache to redirect all requests for the old group to the new group.

To do this, edit the Apache configuration file of whatever Virtual Host you have been serving the WikiServer from. Most of the time, this will be at /etc/httpd/sites/0000_any_80_.conf.

The end of that file probably looks something like this::

#       Include /etc/httpd/httpd_users.conf
#       Include /etc/httpd/httpd_directory.conf
        Include /etc/httpd/httpd_groups.conf
        Include /etc/httpd/httpd_teams_required.conf
        LogLevel warn
        ServerAlias *
</VirtualHost>

Right above these Include directives, simply add the following:

        <Location /groups>
            <IfModule mod_alias.c>
                Redirect 301 /groups/old-group http://your.server.address/groups/new-group
            </IfModule>
        </Location>
#       Include /etc/httpd/httpd_users.conf
#       Include /etc/httpd/httpd_directory.conf
        Include /etc/httpd/httpd_groups.conf
        Include /etc/httpd/httpd_teams_required.conf
        LogLevel warn
        ServerAlias *
</VirtualHost>

There you have it. New Teams Server group, old group’s data.

Caveats

Note that this does not update any of the SQLite databases used to store things like revision history and so forth. These things are, for the most part, not really necessary to update but it would be ideal if the old plist revisions could be changed in there, too. That’s not so much more extra work, really, but I’ve found it typically unnecessary except in fringe cases, so I leave that as an exercise to the reader.

If you do this to your WikiServer and do not update the SQLite databases as well, just be mindful of that fact so that you’re not surprised if something goes wonky down the line.

However, there are some occasions when a text-based response to a complicated question isn’t very helpful because it requires so much extra work to understand. For me, the most common example of this sort of issue has always been in looking at time-based information, and more specifically, calendars. Whenever I’m on my machine, I almost always need to look at a calendar.

In the past, I used to go all the way over to iCal. Sure, I can do this using keyboard shortcuts only, but sometimes all I want is a quick answer to “what date is this upcoming Friday?” In situations like that, I’ve lately begun using the cal command, and my oh my, what a timesaver.

cal is kind of like man for dates. Of course, you can get more info by saying man cal to your prompt. The cal program, installed by default on almost all UNIX-based systems (including Mac OS X), has a ton of useful options. However, most of the time, I don’t need more than a few.

For instance, let’s say I just want a calendar of the current month. I can get get a compact, simple month view instead of going to iCal by saying just cal at the command line:

Perseus:~ maymay$ cal
     April 2008
Su Mo Tu We Th Fr Sa
       1  2  3  4  5
 6  7  8  9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30

Other options let me ask other questions of cal. Easy, simple, fast. I like it.

1. Be certain you have the git-core package installed on your system and that this package includes the git-cvsimport command. You can run git help -a | grep cvsimport to verify this.
2. Be certain you have the cvsps command-line tool installed. This does not come with the git suite of tools, so you’ll need to get it separately. If you’re a lazy Mac OS X user, like me, you can use MacPorts: sudo port install cvsps. Otherwise, get it from the source.
3. Prepare your CVS login information for the remote server before you run git cvsimport. You need to do this so that the git tool will be able to log you in to the CVS server automatically. The command for this looks like:

   CVSROOT=:cvs-login-method:cvs-user-name@cvs.server.name:/path/to/CVS/root cvs login

   For example, if you’re pulling code from the anonymous CVS server that runs on Drupal.org, you might use this: CVSROOT=:pserver:anonymous@cvs.drupal.org:/cvs/drupal-contrib cvs login. This command will prompt you for the password for the user you specified at the server you specified (for anonymous access, the password is almost always anonymous) and will hash this in the ~/.cvspass file for future use by CVS

4. Finally, run the git cvsimport tool, and specify the proper options. Using the Drupal example above, your command might look like this:

   git cvsimport -v -d :pserver:anonymous@cvs.drupal.org:/cvs/drupal-contrib contributions/modules/module-name

   This would login to cvs.drupal.org using the CVS‘s pserver login method, provide the username anonymous and the password you specified in the previous step that is hashed in ~/.cvspass, set the CVS document root to /cvs/drupal-contrib, and pull the code located at contributions/modules/module-name into the current working directory as a git repository.

This works pretty nicely, and creates a git repository just as though you’d created it with git init in the current working directory.

If you get an error that looks like this:

AuthReply: cvs [pserver aborted]: descramble: unknown scrambling method

then you’ve most likely specified the CVS document root incorrectly. Most notably, git cvsimport does not understand a CVS document root wherein the password is specified in the document root URL itself. So, for example, git cvsimport -d :pserver:password:username@cvs.server.name:/path/to/CVS/root code/to/checkout will not work. Omitting the password and the separating colon from the URL should fix it.

First things first. Once you have an idea of what work you want to do, you should give yourself a space to do this work without disturbing anybody else’s work. Do this by making a new, personal branch. Unlike Subversion and some other centralized version control systems, git makes it possible to do make all kinds of changes to your repository, including making branches, and even save those changes without having to republish everything back to the central repository server at each step. In other words, if you thought Subversion branches were “cheap,” you’ll love git’s branches.

Also unlike Subversion, which stores its branches in completely separate pathnames, git keeps all branches in the same filesystem tree separated only with metadata (in .git/refs/remotes for remote branches and in .git/refs/heads for local branches, to be a bit more precise), so you don’t have to create lots of different directories for all your branches (unless you want to). With no branches defined, you’re working in the “master branch,” or “the trunk” by default.

git branch
* master

By saying git branch you ask git to print a list of all (local) branches. The one with the asterisk marks the current branch, the one you’re using at the moment you run the command. Since there are no branches, you’re currently in the “master” branch. But we don’t want to make changes here, we want to make changes in our own private branch, so we’ll make a new one.

git checkout -b new_branch_name

This will create a new branch with the name new_branch_name and immediately switch to it. Notice that you’ve made absolutely no changes to the filesystem itself; only the git metadata has been altered. Saying git branch again will show you the change:

git branch
  master
* new_branch_name

Also note that since we haven’t comitted any changes, we don’t need a commit message (or “log message”) for creating this branch. We’ll add one later, when we need it. Now go ahead and write some code in your new branch. At any time, you can create a new branch in the same way you added the first. Each new branch is created at the HEAD (“latest”) revision of whatever branch you’re currently working in.

git checkout -b another_branch
Switched to a new branch "another_branch"
git branch
  master
  new_branch_name
* another_branch

To switch back to any other branch, simply git checkout that branch again:

git checkout master
Switched to branch "master"

If you want to delete a branch you don’t like, that’s easy too:

git branch -d another_branch
Deleted branch another_branch.

Keep in mind that throughout all of these branch creation and deletion actions, the only thing that’s being altered is the git metadata. That’s why it’s so cheap to create new branches. If you ever have a new idea you’re working on, it’s recommended that you create a branch for it, even if that branch is so short-lived it never gets published.

So you have a new local branch, and you’ve been working as you normally do for a few minutes, creating files, editting them, and so on. Running git status now will ask git to show you the changes you’ve made to your filesystem. If you’ve created any files in new directories that git doesn’t know about, it will simply report that directory. If you’ve made new files in directories git does know about, it will list all those files explicitly.

git status
# On branch cartoon_contests
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#	sites/default/modules/cartoon_contests/
#	sites/default/modules/factiva/.factiva.module.swp
#	sites/default/modules/testfile
#	sites/default/settings.php
nothing added to commit but untracked files present (use "git add" to track)

In the above sample output, I’m working on a new Drupal module for a web site. I’ve created a new directory, sites/default/modules/cartoon_contests/ (note the trailing slash), and I have several untracked files. One is my vim swap file for a different module, .factiva.module.swp, one’s an unimportant testfile, and the last is the Drupal configuration file, settings.php.

The only thing I want to commit is the new cartoon_contests directory, and all the files within it. Like Subversion, I have to tell git that I want to track this directory, which is done simply by saying git add cartoon_contests. Unlike Subversion, future invocations of git status let me see everything that git is going to go in the next commit.

git status
# On branch cartoon_contests
# Changes to be committed:
#   (use "git reset HEAD ..." to unstage)
#
#	new file:   sites/default/modules/cartoon_contests/cartoon_contests.info
#	new file:   sites/default/modules/cartoon_contests/cartoon_contests.module
#
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#	sites/default/modules/factiva/.factiva.module.swp
#	sites/default/modules/testfile
#	sites/default/settings.php

The “Changes to be committed” section of the output is called the staging area, or the index. In this way, I can prepare all the changes I want to commit before I do so, making sure they’re perfect before I actually commit them to the git repository. At any time before I commit, I can make additional modifications, such as git adding more files or directories, git reseting to unstage all (or some) of my changes, etc. git help reset also has a number of handy explanations with examples for different things you might need to do at this point.

If you were using svn:ignore, equivalent functionality exists in git. Simply append file glob patterns, one per line, to the $GIT_DIR/info/exclude file in your git repository. Like so:

echo -e "*.swp\nsites/default/settings.php" >> .git/info/exclude ; git status
# On branch cartoon_contests
# Changes to be committed:
#   (use "git reset HEAD ..." to unstage)
#
#	new file:   sites/default/modules/cartoon_contests/cartoon_contests.info
#	new file:   sites/default/modules/cartoon_contests/cartoon_contests.module
#

If you’ve got many of these, you can use git svn show-ignore >> .git/info/exclude to search through your old Subversion repository and look for any and all ignores, automatically adding them to git’s exclude list. (Checkout Tsuna’s blog entry on learning git for more tips like this.)

Finally, after you’ve done some of your work and you’ve finished staging your changes, you’re ready to commit them to the repository. On the other hand, if you hate what you’ve done and want to undo it all, you can say git reset --hard HEAD to throw away all your local changes. To throw away changes to just a single file, just checkout that file again by saying git checkout filename. This is the equivalent of Subversions svn revert filename command.

Before you actually make your first commit, however, you should properly introduce yourself to git. You don’t have to do this because git will try to figure out who you are by itself (details explained in detail here), but you probably should at least create global defaults for your user (which git will store in ~/.gitconfig). If you want to, you can also create per-repository defaults (which git will store in .git/config), or even system-wide defaults for all users of this computer (which git will store in /etc/gitconfig). To do so, say this:

git config --global user.name "Your Full Name"
git config --global user.email "you@example.com"

This will create the file ~/.gitconfig if it doesn’t alreay exist and will write your name into it. You can also just edit the file directly yourself instead of using git config commands.

Like Subversion, committing will create a saved, fixed point-in-time that marks the changes you have made to your files. Like branches, commits are also very cheap in git, so go ahead and commit at any time you like. Remember to stage your files (by git adding them), and then just git commit.

git add filepattern
git commit -m "My very frst git commit!"
Created commit ef483c1: My very frst git commit!
 5 files changed, 58 insertions(+), 12 deletions(-)

If you forget to specify a commit log message (-m "log message") on the command line, or if you want to enter a multi-line commit log, git will prompt you for it in your favorite $EDITOR. You can view a history, including all the log messages, for your project with git log. You can even the view logs in a number of pretty formats. Check git help log for more information.

If you want to change anything about the commit you just made, such as the author, you can just run git commit again with the --amend flag added to the command. Notice the typo in the commit log? Fixing it is really easy:

git commit --amend -m "My very first git commit!"
Created commit 88602f6: My very first git commit!
 5 files changed, 58 insertions(+), 12 deletions(-)

Finally, with your new code commited to your local git branch, it’s time to share that code with your colleagues who are (for some inexplicable reason) still using Subversion. This is also extremely simple. Just say git svn dcommit. (That’s right, dcommit, not just commit. I don’t know why….)

Not sharing your changes via Subversion, but with a patch instead? git diff -p will generate patches for you. See git help diff-files and look for the “GENERATING PATCHES WITH -P” section.

If you found this helpful, you may also enjoy an alternative tour for beginners from Carl Worth.

Using Mac OS X, first make sure you have the git core installed as well as the git-svn components. Unlike other version control systems, git is not one monolithic program but a collection of smaller utilities, and the Subversion conduits are a subset of these utilities. The git-Subversion utilities themselves depend on having the Perl::Subversion bindings installed for Perl 5.

For the lazy MacPorts user, simply run:

sudo port install git-core +svn

This will install the git-core, the git-svn packages, as well as all the documentation for git and any required dependencies you don’t already have. Note that the documentation (man pages) is installed in /opt/local/man, which may not be in your default $MANPATH, so be sure to add that directory if man git returns a “No manual entry for git” error.

Alternatively, if you don’t want to use MacPorts, you can download a pre-compiled Mac OS X binary that includes git, the git docs, and the git-svn package from the git web site that comes complete with a standard GUI installation procedure.

Next, initialize a new git repository that tracks a remote Subversion one. This allows you to work privately using git, but to collaborate with other people who are using Subversion. Running

git svn init svn://my.svn.server/path/to/svn/repo workingcopy

will give you an empty git working copy named workingcopy configured to use the remote Subversion repository at svn://my.svn.server/path/to/svn/repo. This step is analogous to the need to run svnadmin create repo_db, to initialize a new repository database (where all the file versioning information will be stored). Unlike Subversion, git’s distributed database means that the working copy itself is also the location of the repository database, so there’s no need to deal with two filesystem paths anymore.

Next, change directory to your working copy, and run

cd workingcopy
git svn fetch

to populate your new, empty git working copy (and repository) with all the files from the remote Subversion repository.

Now that you have filled your git repository with a lot of data, if you want to, you can now also save a significant chunk of disk space by repacking that data into a more single, efficient, native packed git archive format (a .pack file). The git-repack -a command is used to do this, and its manual page says:

Instead of incrementally packing the unpacked objects, pack everything referenced into a single pack. Especially useful when packing a repository that is used for private development and there is no need to worry about people fetching via dumb protocols from it.

According to some sources, this can turn a 353MB Subversion repository into 31MB of git pack. Say:

git repack -a -d -f
Generating pack...
Counting objects: 284
Done counting 4475 objects.
Deltifying 4475 objects...
 100% (4475/4475) done
Writing 4475 objects...
 100% (4475/4475) done
Total 4475 (delta 1876), reused 0 (delta 0)
Pack pack-a115c320ff9c9968248bd250bdfea3110d0f0c1a created.
Removing unused objects 100%...
Done.

to perform the compression. For even greater savings, increase the compression by stipulating a high --window option, such as git repack -a -d -f --window=100. (--window defaults to 10—the higher the window, the harder git-repack will try to compress your stuff.) This turned a 36MB Subversion repository in one of my projects to a 20MB git pack. As always, your mileage may vary.

Congratulations, you have transformed your old Subversion repository to a git repository. All that’s left to do now is to get to the coding. Perhaps start by making a local (cheaper than cheap!) git branch….

That was easy, right? Most things in git are, in fact, that easy. If you’ve never used other version control systems before, be grateful. If you have, you can breathe a sigh of relief. Still not convinced? Don’t take my word for it…ask Linus Torvalds (or Randall Schwartz).

While playing with my Desktop Backgrounds (computer “wallpapers”) the other day, I ran into an issue like this. I wanted to make sets of wallpaper images that overlapped one another. This is easily doable with Apple’s iPhoto, since all iPhoto albums appear as Desktop Background sets by default. I could simply put together a bunch of iPhoto Albums containing the images I wanted, including overlapping images, and be done with it. iPhoto takes care not to duplicate the photo into every album so space on my hard drive would not have been wasted.

But what if I didn’t want to use iPhoto? If I just have a bunch of image files on my disk in folders, how can I make overlapping sets? The solution is as simple as a Unix symbolic link—with a catch: only a symbolic link with an absolute path would work.

At first, I tried to simply use a standard Mac OS X “alias,” but that didn’t work. My MacBook Pro wouldn’t follow the alias file to the original file when trying to display the wallpaper. Then I tried a simple symlink from one file to another, but again it didn’t work.

Luckily, then I tried dragging the file into my terminal window, which automatically listed the full POSIX pathname of my image. This, surprisingly, worked. So it turns out that the Desktop & Screen Saver System Preference pane requires an absolute path to be specified if it’s to follow symlinks for creating wallpaper sets organized by folders. Go figure.

Thankfully, there’s an easier way. Especially if you’re accustomed to “old-school” development workflows that you learned through, say, PHP or perhaps Perl scripting, then ditching Rails and starting purely with Ruby might be the way to go. Doing things on your own without the “magic” of Rails will let you learn Ruby in a simple, yet fully-featured environment, much like the way you might have learned classic CGI programming.

Specifically, by using eRuby, the Embedded Ruby interpreter, you can make Ruby web development feel just like PHP development. Install it, upload an HTML file with some embedded Ruby code in it, and—voila—you have a single-page web application.

Why this tutorial?

Hivemind’s Getting Started with Ruby on the Web is an excellent resource that described this process from start-to-finish for Windows users. In fact, it was after finding that article that I realized web development with Ruby didn’t have to be governed by Rails and, being a web developer like the authors of Hiveminds, I wanted an easy way to use my Ruby knowledge on the Web.

However, I couldn’t find any similar tutorial or walkthrough for those of us who use Macs. Even the Linux walkthroughs are lacking (or at least my Googling skill for them is). After spending a little while successfully getting eRuby set up on my Mac running Mac OS X 10.4 Tiger, I thought I’d share what I learned with others, so I wrote this article. Its content mirrors the one from Hiveminds to a great degree but focuses on getting eRuby installed on Mac OS X instead of Windows (obviously). I imagine these instructions should be pretty similar if not identical for any other *nix-like platform.

The Setup

Getting eRuby installed and configured with your web server is actually pretty straightforward and won’t take an experienced web developer much time at all. All we’re going to need is the Apache web server and the Ruby programming language to start, both of which are already pre-installed on Mac OS X 10.4 Tiger. (And in Mac OS X 10.5 Leopard, Apple is even shipping new Macs with Ruby On Rails preconfigured, but that’s a subject for another time.) However, Apple typically ships an older copy of Ruby than is readily available, so you might choose to use a newer version of Ruby that you download yourself.

The easiest way to get a copy of eRuby (and an updated Ruby, if you want it) is with a package manager such as MacPorts. With MacPorts installed (which is a simple, standard package installation downloadable from the MacPorts home page), simply run

sudo port install eruby

from your terminal to download the latest stable Ruby and eRuby in one fell swoop. After this runs, you’ll have a new Ruby in /opt/local/bin/ruby and an eRuby in /opt/local/bin/eruby. Alternatively, of course, you could compile and install eRuby yourself. Instructions for that are on the eRuby homepage.

As a quick test to see if everything is installed and working correctly, create a plain text file with your favorite editor that contains the following code:

Hello world! The time is now <%= Time.now %>.

All this does is print out a “Hello world!” greeting and then announces the current time (as reported by your system clock, of course).

If you named the above file helloworld.rhtml then you can feed this file to the eruby interpreter as follows:

eruby helloworld.rhtml

Note that the file extension doesn’t actually matter. You could use .erb, if you like, or any other arbitrary file extension. The file extension is, as you’ll see later, only used for telling Apache which kind of file it is. So once you come up with file extension you like, stick with it. The standard file extensions are either .rhtml or .erb, so I’d recommend using one of those. (I prefer .erb, myself, since I don’t necessarily only want to create HTML pages. I might want to create XML documents like news feeds or even XML databases, so .erb seems a more reasonable file name extension, though most of the eRuby documentation uses .rhtml.)

If things worked properly, you should see output similar to the following:

Hello world! The time is now Wed Oct 24 21:58:54 -0400 2007.

Serve it up!

Now that eRuby is installed and working, the next step is to set it up as a CGI so that when your web server (Apache, in my case) fetches the file it will first feed it through eRuby for processing before sending it back to the browser. The simplest way to do this is to configure Apache to use a specific directory somewhere on your filesystem as a CGI directory—that is, a directory whose contents are all treated as CGI programs. You do this via Apache’s ScriptAlias directive.

Conveniently, standard installations of Mac OS X already come with a CGI directory. It’s located at /Library/WebServer/CGI-Executables. This directory is just like any other, except that Apache treats it specially. It treats it specially because Apache’s configuration file, /etc/httpd/httpd.conf contains a line of text that reads as follows. (On a standard installation of Mac OS X client, this is line 671 of the file.)

ScriptAlias /cgi-bin/ "/Library/WebServer/CGI-Executables/"

This line basically just tells apache, “Treat every file in the /Library/WebServer/CGI-Executables directory as though it were a CGI program.” So, since that’s how we’re treating eRuby, we’re going to want to put our eruby binary in that directory. However, we don’t actually want to make copies of the binary. Having too many copies of a program on your system can easily get confusing. So instead, we’ll make a symbolic link (or an alias to use the classic Mac terminology) to the binary.

ln -s /opt/local/bin/eruby /Library/WebServer/CGI-Executables/eruby

With our eruby CGI executable in the proper place for Apache to find it, we can now tell Apache which files we want it to send to this program. We do this using the AddHandler and Action directives.

AddHandler simply tells Apache to treat files with a certain extension in a certain way. For instance, you can say something like, “Treat all files ending in .php as PHP scripts.” Similarly, you can (and will) tell Apache, “Treat all files ending in .rhtml or .erb as eRuby CGI pages.”

To do this, we add the following line to our /etc/httpd/users/username.conf file somewhere inside the <Directory> and </Directory> block. Naturally, replace username with, of course, your Mac OS X user’s short name:

AddHandler rubypage .erb .rhtml

Again, this just tells Apache that files ending in .erb and .rhtml should be treated the same way other rubypages are treated. But how are eRuby pages supposed to be treated? They’re supposed to be handed off to the eRuby program, of course, so we need an Action directive that applies to the rubypage handler we just defined.

Right below the above line, add the following line, too:

Action rubypage /cgi-bin/eruby

This line tells Apache that any file that it is treating as a “rubypage” file should be sent to the program at the URL accessible at /cgi-bin/eruby, which we’ve defined earlier to be our symbolic link to the eruby interpreter.

Your completed user-specific Apache configuration file should now look something like this:

<Directory "/Users/username/Sites/">
…
    # Let eRuby files get parsed through the proper CGI binary
    AddHandler rubypage .erb .rhtml
    Action rubypage /cgi-bin/eruby
</Directory>

Are You Being Served?

That’s it, everything should now be in place. To make sure, move your earlier helloworld.rhtml file into your user’s Sites folder, and now simply point your browser to http://localhost/~username/helloworld.rhtml (again, replacing username with your Mac OS X user short name, of course). Congratulations, you’re serving up Ruby-coded applications via eRuby.

Additional Reading

This is just the set up, of course. Now the real fun begins: learning Ruby. Here are some suggested places to start.

• Programming Ruby: Ruby and the Web — A chapter out of the free book Programming Ruby all about Ruby as a CGI programming language.
• Hiveminds: eRuby: How to use Ruby CGI and ERB templating for Web Pages — a quick’n'dirty article explaining the basics of using Ruby on web pages in classic CGI style where program logic and HTML templates are kept separated, as opposed to classic PHP style, where the two are mingling together on one page.
• COOLNAMEHERE: Simple Ruby CGI — Like the Hiveminds example above, another quick’n'dirty Ruby CGI how-to.
• cgi: Ruby Standard Library Documentation — Ruby’s standard library contains a rich set of features, and the CGI objects are among them. They are well-documented in the official Ruby Documentation Project.

Naturally, feel free to leave comments if you have more resources that you found helpful.

But then today I was glossing over Apple’s featured features list and I got even more excited. There are the usual, largely meaningless, fluff updates that are nice for Joe Schmo or his mother, but that power users simply don’t care about, like the new iChat support for animated buddy icons, but the list is also chock-full of really cool, really useful features.

What’s interesting is that a good deal of these features aren’t really new features at all. For instance, if you knew how to manipulate the NetInfo database on your Mac, you could already share any folder via Apple’s “Personal File Sharing” feature. (Here’s a Mac OS X Hints hint explaining how to do it.) In Leopard, however, Apple claims that this functionality is now integrated straight into a folder’s Get Info… window. If it works as smoothly as Apple claims, this is finally going to bring Mac OS X (client) into decent competition with Windows XP Professional in terms of GUI-level power-user features.

However, while all of these features are really cool, here’s a list of the ten geekiest features I will probably absolutely love, for one reason or another.

• Ruby on Rails, out of the box — The hot thing in web development right now is Ruby on Rails. Macs have already been the best personal desktop and web development platform because they have built-in support for the Apache web server and a host of other features, but now they will come with a ready-to-roll installation of Ruby on Rails, sporting Mongrel and (better yet) Capistrano! Specifically with the addition of Capistrano, which is terribly undersold as simply a Ruby on Rails deployment platform, these UNIX-y “toolbox” items are bound to make Macs that much more useful right out of the box.
• Safari’s full history search — As their recent public partnerships with Google have shown, Apple is very clearly invested in search technologies. Spotlight gets a huge number of improvements in Leopard, but none which I think are going to be more useful to more people than this one: spotlight searches on the full text of each web page in your visited history list. That’s just awesome. Also awesome: using spotlight as a calculator and as a dictionary, which also shows just how Google-like Apple is trying to be. (Google also lets you ask it arithmetic questions and a dictionary.)
• Wikipedia articles in Dictionary.app — I love Wikipedia because it’s one of the fastest ways to get (relatively) reliable information quickly. Now that Dictionary.app has built-in integration with Wikipedia, imagine the possibilities for getting that knowledge instant-gratification craving fixed. Apple has not yet announced this capability, but I can easily envision a scenario where all Cocoa text fields are instantly “wikified” (with text that matches Wikipedia articles highlighted) much in the same way that current Cocoa text fields allow you to right-click on a misspelled word and have it corrected by Dictionary.app.
• Application-based firewall — In classic Apple fashion, functionality that was previously available via third-party additions is now available from Apple itself. In this case, I have to wonder how well Apple’s updates to its firewall will obviate the need for Little Snitch, which is basically an application-based firewall, too, and a good one at that.
• Built-in guest log-in account — If you’re as paranoid about security as I am, you’ve already created a special, limited-access user on your system (called Guest or Visitor or whatever) and whenever friends are over, you tell them to use that account instead of your own. Now in Leopard, Apple has gone through the trouble of setting this up for us already. A small change that is going to have a big impact.
• Scriptable System Preferences & applications — With AppleScript, you can automate the things your computer does with scripts, as long as those things are “scriptable.” In previous versions of Mac OS X, huge gaping holes of what things shipped by Apple were scriptable existed, causing me (personally) some really annoying headaches. AppleScript GUI scripting helped me get around many of those roadblocks, but now it seems Apple is finally filling in some of the most notorious gaps in this functionality with scriptable System Preferences. Yay!
• Automator workflow variables — Automator brings the power of AppleScript I just mentioned to more people with a completely graphic programming environment. There is no need to open up a text document and write AppleScript code because Automator lets you create a script (called a Workflow in Automator jargon) using your mouse by dragging and dropping actions into the order you want them to be performed. It’s very slick, but until now it’s been very limited. With Leopard, Apple is beefing up Automator so that it includes things like variables, basic programmatic capability that was sorely lacking before. (Also majorly cool: a command-line utility to access Automator!)
• Finder.app’s path bar — Every serious Mac user knows that the Finder needs a lot of help. Now, it’s getting some. Something the Windows Explorer has had forever (as had every desktop environment for Linux, of course) is a visual cue to show you where in your filesystem tree a given folder is located when you are viewing said folder. Now the Finder gains this capability (though Apple’s description implies that it’s going to be off by default) with what Apple is calling a “Path Bar”. Finally!
• Cocoa and scripting bridges — Even though no one really seems to know about it, it has long been possible for languages other than AppleScript to do things like send Apple Events to Mac OS X applications. Specifically, Ruby and JavaScript, two of the most well-known web development languages in existence, can already do this with a single ScriptingAddition (OSAX). But now Apple is making this functionality a central feature and fully extending it to their Objective-C (and Cocoa) language and applications such as Xcode and Interface Builder. This means people like me will have a shallower learning curve before we’re able to create full-fledged, native Mac OS X applications. Now that’s exciting!
• Xcode 3 refactoring — This is something you kind of have to see to believe. I got the opportunity to see it demoed at Apple’s Leopard Tech Talks last year and I was really excited by it. With the new Xcode, Apple’s development IDE, you can do away with find-and-replace searches for things like renaming functions because Xcode understands what parts of your code are what structures and, when you tell it to “change the function named myFunction to myNewFunction,” it’ll only find-and-replace function names instead of every instance of the string “myFunction.” That’s pretty big, and if it were available for more languages, it’s almost enough to make me ditch vim.

So there you have it. Ten features you might not have already known about that are some of the most promising features I can see in Leopard. And I didn’t even get into Wide-Area Bonjour, which could make services like DynDNS or No-IP a thing of the past (and which I still want to learn more about), or the new Terminal application (finally with tabs!), or even the multiple user certificates for S/MIME encrypted email.

Note: One of the least known security features available on Mac OS X is also possibly one of the best, and the simplest. Evidently, all Intel-based Macs are shipped with the XD (aka. NX, aka. DEP) bit turned on—and thankfully there doesn’t seem to be any way for users to turn it off. However, this isn’t a silver bullet and if you want to learn why you should check out this excellent Anandtech article: A Bit About the NX Bit.

Anyway, the experience of moving one’s data from one machine to another is always a bit of a hassle. There are so many little things you’re sure you’ve forgotten, preferences you don’t want to have to recreate, and small gotchas that, if you’re not careful, will mean you’ve lost something like your old email or your contacts—sometimes for good.

Thankfully, with a remarkably few easy planning steps, a lot of this trouble can be avoided all together. Much of this is thanks to the brilliant architecture of Mac OS X, which does a superb (though not perfect) job of separating data from applications. The other major benefactor of this kind of easy transition is server-based data stores, or hosted services (even if those services are ones I’ve hosted myself).

So here’s what I wanted to move over from my old mac to my new one:

• Personal documents, pictures, movies, and music, etc.
• Preferences for all applications, such as my dock layout, the Finder’s and Mail’s toolbar button arrangements, and so on.
• PIM databases, such as my Address Book contacts, iCal calendars, my email, and the like.

This could really be a much longer list, especially if I were going to name every single item of personal importance. Nevertheless, each item falls into one of these three main categories of stuff that I want to move.

The first time you turn on a new Mac it will ask if you want to transfer your data from your old computer to your new computer. It does this using the Migration Assistant application located in your Mac’s /Applications/Utilities folder. While this works well for the most part, I’ve seen this program fail one too many times—especially for cross-architecture, PPC to Intel, transfers—that I’m simply not a fan of using it. Besides, as you’ll see in a moment, it’s trivially easy to do all of that yourself.

Of course, all of your personal information should be somewhere in your Home folder. You could just copy the entire directory from one Mac to the other, but you’ll probably end up deleting some (albeit probably non-critical) items from your new machine. When you drag-and-drop in the Finder, the Finder first removes anything of the same name on the destination as the source of the transfer. This is not what you want.

So here’s how I dealt with moving each type of item, after ensuring I had a good backup:

Transferring personal documents

rsync is a godsend for any situation where you want to ensure that you’re copying files from one location to another. It compares the contents of folders and, by default, only adds missing items to the destination while leaving everything else alone. It also overwrites identically-named files on the destination with files from the source.

To use rsync, I first needed to turn on the destination Mac’s “Remote Login” feature; this turns on the SSH server. (Find this in System Preferences → Sharing → Services.) Next, simply issue this relatively simple command at the command line, from your home directory:

rsync -aEve ssh Documents UserShortName@DestinationMac:Documents

This tells rsync to copy files from the local machine’s Documents directory to the DestinationMac‘s Documents directory in archive mode (the -a option), which preserves numerous file meta-data such as its modification times and access permissions, along with the file’s extended attributes and resource forks, if any exist (-E), reporting verbosely (-v) as it does so, executing over SSH (-e ssh). The UserShortName is the DestinationMac‘s user short name, obviously, and the DestinationMac is either its hostname (such as mynewmacintosh.local) or its IP address (such as 192.168.0.2).

I repeated the above step for the Desktop, Movies, Music, Pictures, Public, and Sites folders. Most of my music is actually on an external hard drive, so that was of course a cinch, and rsync took care of the rest.

Transferring preferences

Mac OS X stores its user-specific preferences in each user’s home Library/Preferences folder. The set up is elegant and beautifully implemented. Just like the folders above, I could have simply rsync‘ed my preferences over, but I wanted to be more selective. So I simply scrolled through the list finding the preference files for the applications I still use and copied them over. I used the Finder’s drag-and-drop, but you could use whatever method you want.

These files are named in reverse-DNS style notation, so for instance, my Mail preferences were stored in the file com.apple.mail.plist.

This was a super-simple step.

Transferring PIM databases

Like preferences, user-specific data is often stored in that user’s home Library/Application Support folder. Inside that folder is a folder for each application that has some data to store. All of iCal’s information, for example, is thus stored in your home Library/Application Support/iCal folder.

rsync to the rescue again:

rsync -aEve ssh Library/Application\ Support/iCal "UserShortName@DestinationMac:Library/Application\ Support/iCal"

In this case, because of the space in the Application Support folder’s name, the quotations and the backslashes are required to help rsync find the proper folders. Other than that, the command is the same as before.

I selectively copied over all the data from the applications I wanted to keep and, voila, the next time I opened iCal or Address Book, my data was there.

The Mail application and Safari both have special ways of handling their information, however, so their datastores couldn’t be found in the Application Support folder. Instead, they keep their folders directly inside the Library folder named (predictably) Mail and Safari, respectively.

Also, since I use IMAP for email and shared calendars in the form of .ics subscriptions whenever possible, the natural sync process for these databases saved a lot of time.

Most other programs in non-standard locations could be found with a little digging inside my Library folder.

It should be noted that for this to work reliably, you should always move all the preference files along with the Application Support directory for any program whose Application Support directory you’re grabbing!

Cleaning up

All told, the whole thing took maybe two hours and most of that time was spent transferring data. It was really quite painless. Because this was a surgical transfer, instead of a full “migration,” there’s very little to clean up.

However, for some reason, Apple’s .Mac Sync service had quite a bit of trouble keeping up with my new Mac’s new additions. The fix was simple enough: unregister every sync’ed computer, then reset all of the .Mac Sync data with the data on the new Mac. Then, re-sync each device either merging its data during its syncing process or replacing the data on it with the new, known-good data on .Mac. (Again, make sure your backup works before you do this.)

And that’s pretty much it. Now I have a new iMac, but it looks exactly like my old one on the screen. :)

In addition, many times you want to run multiple commands with only slight variations. Perhaps you are benchmarking your Web server with ApacheBench and want to change some of its settings. Whatever you are trying to do, many shells actually offer a full suite of command-line editting shortcuts.

Some of them are built into the shell itself and others are configured elsewhere. Unfortunately, different distributions and different operating systems sometimes don’t have the same shortcut configuration. There are lots of lists out there for Bash keyboard shortcuts for Linux, and many of these work on Mac OS X, but they don’t all work.

So here’s a list as complete as I can make it of the default keyboard editting shortcuts configured on a Mac OS X machine running bash. An extra goodie is that some of these movement shortcuts work in any Coacoa text field, which means you don’t have to keep switching between bash’s command line movement shortcuts and the +arrow keys shortcuts anymore.

CTRL+a

Move cursor to beginning of the line. (I think of this as anchoring my cursor to the start.)

CTRL+e

Move cursor to the end of the line.

CTRL+k

Delete everything from under the cursor to the end of the line. (I think of this as killing the rest of my line.)

CTRL+f

Move forward one character. Identical to →.

CTRL+b

Move backward one character. Identical to ←.

ESC+f

Move forward one word.

ESC+b

Move backward one word.

CTRL+u

Delete everything from under the cursor the beginning of the line.

CTRL+w

Delete from under the cursor to the beginning of the word.

CTRL+r

Recall previous commands by searching for them.

CTRL+t

Transpose (swap) the two characters before the cursor with one another.

ESC+t

Transpose (swap) the two words before the cursor with one another.

CTRL+y

Paste the most previously-deleted string. Basically a sort of command-line editting “undo.”

CTRL+z

Stop the current process and send it to the background.

CTRL+c

Send an SIG_HUP to the current process. The net effect of doing this on the command line is that you cancel your current command and are presented with a blank new line.

CTRL+d

Send an end-of-file special character to the current process. Doing this at the command line is identical to closing your terminal window.

CTRL+p

Recall previous command executed. Identical to ↑.

CTRL+d

Forward delete.

CTRL+h

Backspace.

CTRL+j

Carriage return. Identical to hitting the return key.

CTRL+m

Newline. Identical to return.

CTRL+l

Repaint screen. This is useful if a program’s output is overwriting some text on your terminal. The effect of doing this on a command line is that you clear the screen. Note than in Apple’s Terminal.app, you can also press +K to clear the screen.

CTRL-x + CTRL-x

Mark current location in line and jump to beginning of line or second mark if defined. Repeat to jump to between both marks.

CTRL+v

Insert next character verbatim. This is how you escape control sequences. For instance, to literally send a

CTRL+[

Identical to ESC.

ESC+c

Capitolize word under cursor and move to next word.

ESC+u

Uppercase word under cursor and move to next word.

ESC+l

Lowercase word under cursor and move to next word.

ESC+.

Insert last word from previous command after cursor.

TAB

Auto-completes file, folder, and program names.

ESC-?

List the possible completions

CTRL-x /

List the possible filename completions

ESC-/

Attempt filename completion

CTRL-x ~

List the possible variable completions

ESC- ~

Attempt username completion

CTRL-x $

List the possible variable completions

ESC-$

Attempt variable completion

CTRL-x @

List the possible hostname completion

ESC-@

Attempt hostname completion

CTRL-x !

List the possible command completions

ESC-!

Attempt command completion

ESC-TAB

Attempt completion from previous commands in the history list

Note that this list assumes you’ve not reconfigured any of these shortcuts. Again, they’re the default keyboard assignments. Furthermore, they’re the default keyboard assignments for the default command-line editting mode, which is emacs mode. If you’re familiar with the vi editor, you can change this to use all the cursor movement and deletion keys you’re familiar with by saying set -o vi to your shell.

External references

Here’s some other people’s lists. They’re basically the same, but what the hey.

• BASH Command-Line Editing at lxpages – Also has a listing of vi-mode key shortcuts.
• The BASH Reference Manual for Command Line Editing
• Bash Features – Command Line Editing – discusses even more commands that Readline can do for you when using bash.

This improved version of the post-commit hook does the same thing as the last one (that is, it logs into your web server over SSH with the given user and password, and yes, I’m aware of the scariness of embedding a password in such a way, so you should really set up SSH to use public keys for authentication for this), except now it also produces useful output.

Here’s the same script as before, but improved:

#!/usr/bin/expect -f

#
# AUTHOR: Meitar Moscovitz 
# DATE  : Thu Jun 21 16:32:42 EDT 2007
#

set HOST my.web.server
set USER someuser
set PASS xxx

# the working copy we're going to update
set WC /path/to/working/copy

# the path to the svn executable on the remote web server
set SVNBIN /usr/local/bin/svn

# our network is slow, set a long timeout
set timeout 30

##### DO NOT EDIT PAST THIS LINE! #####

# POST-COMMIT HOOK
#
# The post-commit hook is invoked after a commit.  Subversion runs
# this hook by invoking a program (script, executable, binary, etc.)
# named 'post-commit' (this file) with the
# following ordered arguments:
#
#   [1] REPOS-PATH   (the path to this repository)
#   [2] REV          (the number of the revision just committed)
#
# Note that Subversion does not provide this program with an environment
# of any kind. That means this program lacks a current working directory,
# a home directory, a $PATH, and so on.

set REPOS [lindex $argv 0]
set REV [lindex $argv 1]

# Define error codes
set E_NO_SSH     1 ;# can't find a usable SSH on our system
set E_NO_CONNECT 2 ;# failure to connect to remote server (timed out)
set E_WRONG_PASS 3 ;# password provided does not work
set E_UNKNOWN   25 ;# unexpected failure

# find the SSH binary on our system
if {[file executable /usr/bin/ssh]} {
	set SSHBIN /usr/bin/ssh
} elseif {[file executable /usr/local/bin/ssh]} {
	set SSHBIN /usr/local/bin/ssh
} else {
	send_error "Can't find a usable SSH on this system.\n"
	exit $E_NO_SSH
}

spawn $SSHBIN $USER@$HOST $SVNBIN update $WC

expect {
    "continue connecting (yes/no)? " { send "yes\r"; exp_continue; }
    -nocase "password:" { send "$PASS\r"; }
    timeout {
        send_error "\nWe have timed out after $timeout seconds while trying to connect to $HOST!\n";
        exit $E_NO_CONNECT;
    }
}

expect {
	-nocase "password:" { ;# if we are asked for the password again, then we have provided the wrong password
		send_error "\nCan not log in to $HOST because the password provided for user $USER has been rejected.\n";
		exit $E_WRONG_PASS;
	}
	-re "revision (\[0-9]+)." {
		if {$REV == $expect_out(1,string)} {
			send_user "\nSuccessfully updated $WC on $HOST to revision $REV.\n"
		} else {
			send_user "\nUpdated repository to revision $expect_out(1,string), but svn reports that we are at revision number $REV.\n"
			send_error "CAUTION: Repository updated to revision $expect_out(1,string), but committed revision $REV.\n"
		}
	}
	default {
		send_error "An unexpected error has occured. The process at spawn ID $spawn_id has produced the following output:\n"
		send_error $expect_out(buffer)
		exit $E_UNKNOWN
	}
}

Still, however, the problem was not exactly straightforward. We needed to execute a svn update command on a server other than the server on which the Subversion repository was being hosted. Shell scripts are the obvious choice for command-line automation in UNIX, but they don’t deal with interactive commands very well. So instead of writing a shell script, I wrote an expect script.

This expect script is really basic. There’s a better one in a future post on this topic.

#!/usr/bin/expect -f

#
# This expect post-commit hook connects to staging-webserver
# and updates the working copy hosted there to the latest checked-in code.
# This means that whenever code is committed to the repository, the web site hosted
# will always be running the latest version of the code.
#
# AUTHOR: Meitar Moscovitz
#

# POST-COMMIT HOOK
#
# [...]
#

set REPOS [lindex $argv 0]
set REV [lindex $argv 1]
set HOST staging-webserver
# Use update-user to log in to staging-webserver
# to update the working copy, but this can probably be improved so as not
# to expose this user's password.
set USER update-user
set PASS update-user's-password

spawn /usr/bin/ssh $USER@$HOST svn update /path/to/web/site/directory
expect "continue connecting (yes/no)? "
send "yes\r"
expect "password: "
send "$PASS\r"
expect eof

There’s one really tricky bit to this script, which is the understanding that when Subversion runs the post-commit hook, no environment information is passed to this script. As a result, there is no home directory or path information set for this executable. This is why everything is defined using absolute paths. Also, because there is no home directory for update-user, the expect script will always be prompted by SSH to re-verify the server’s identity. So rather than just expecting “password: “, we always expect “continue connecting (yes/no)? ” and say yes, and then send our password.

Note, of course, that update-user should be a user with limited access to the system, yet enough so that he may update the working copy on the web server. I’m sure there is probably a more secure way of doing this as well, so any sort of feedback on securing this or scaling it would be welcome.

Shells and Terminals: What’s the difference?

The very first thing anyone thinks about when they think UNIX is the command line interface (or CLI, for short). The funny thing about the CLI is that it’s actually a program in and of itself called a shell that takes what you write on the command line and basically translates it to the computer so it will do what you tell it to do.

So the computer (or the OS) is really only accessible through this shell. This is fairly straightforward until you realize that the shell is usually only accessible through another program, commonly called a terminal, which on Mac OS X is called Terminal.app. Therefor the terminal is really a window to a shell which is another window to the OS, which then talks to the computer for you. So when people say they want to learn to use the command line, you can understand the confusion that might cause. What shell do they want to use? And on top of that, which program are they intending to use to access their shell? Even on the Mac, there are at least two built-in terminals you can use (Terminal.app and XTerm) and maybe half a dozen shells.

As far as terminals go, Terminal.app is the most accessible (and you can find it in your Utilities folder in your Applications folder on your startup disk). My favorite resource for getting started with Terminal was MacDevCenter‘s excellent article, Learning the Mac OS X Terminal, the title of which makes a lot more sense after understanding the previous two paragraphs. That article will show off some of Terminal’s impressive features that make a GUI environment complement the command line environment nicely.

The Command Line: Speak and it will be so

The command line itself is, of course, yet another subject entirely. Thanks to its spartan appearance, I think the best way to learn it is to just use it. If you’re feeling brave and have a spare computer lying around, take the leap and install a Linux or a BSD distro, but run it all from the command line (which is simply a matter of changing the active runlevel on most UNIX-like OS distros). This will effectively boot you into a shell, with no GUI coating. For Mac users who want to stay closer to home, you can do the same thing by logging into Mac OS X as the >console user (with no password). This strips Mac OS X of its Aqua interface and drops you into a login shell.

The important thing to remember is that, while in a shell, you can do everything you can do in a GUI mode. You can browse the web, edit and view documents, and even read and compose email. Of course, there won’t be any pictures, but the point remains that these things can be done. A week of running your computer in a shell and you’ll feel more comfortable than ever at the command line.

But while you’re there, you’ll want to acquaint yourself with several commands you can use to get help. The most important is the man command, which will let you browse the documentation for any command you want to use. You can even browse the documentation on the man command itself by typing man man (and I strongly recommend that you do that).

Another command you’ll want to get familiar with is apropos. If you type man apropos, you’ll get a very description of what that command does. (I won’t spoil the treat for you.) A similar command to apropos is whatis. All these commands are just programs like any other, absolutely identical to the ones you’re already using with the sole exception that instead of outputting pixels to a screen, they output characters to a shell.

On the web, one of my oft-referenced resources for learning the basics of a shell environment (and thus the command line) is TuxFiles. It’s geared towards Linux users, but the things you can learn there apply to all UNIX-like operating systems. Another one is LinuxCommand. Check them both out.

Inside the Shells

By now you’re hopefully running your computer with a terminal window constantly open in the background, or at least you’re excited about trying it. So what’s next? Learn a new shell! By default, most OS‘s will use bash, an abbreviation for the Bourne Again Shell, which itself is based on the Bourne Shell (get it, get it?), written by Steve Bourne.

Different shells all do the same thing, which is enable you to speak with (by typing to) your computer and have it do your bidding, but they understand different dialects so you need to speak to them in slightly different ways. There are dozens of shells you can use, and which one you ultimately choose depends on the features and syntax of each. I spent a good deal of time reading about them all on Wikipedia before I decided to scrap the whole ordeal and stick with Mac OS X’s default bash shell. However, I learned a lot along the way, and that was really the point.

Speaking Command Line is like learning a new language

Ultimately however, it’s not about the shell or terminal you use, but about the things you can do. This, unfortunately, has to come with time. It’s exactly like learning a language. Every command you learn is akin to learning and using a new verb. The more you use a new language to communicate, the better you’ll become at speaking it. The same is true of the command line.

Keep your terminal program open 24/7/365, and whenever you’re doing something, ask yourself if you could do the same thing on the command line. If you’re not sure, Google it, and if the answer is yes, do it that way the next time. This is especially useful when what you’re doing is more than one thing. It takes a dozen or so clicks to open a file in TextEdit, copy it to the clipboard, past it to a new document, then add some text to the end of the new document, and finally save it again, but all of that can be done by typing a single line in a terminal. (Hint: use your shell’s piping and I/O redirection features.)

The more you immerse yourself, the better you’ll become. If you’re really dedicated, I’ve heard nothing but great things about the book Learning Unix for Mac OS X Tiger, which describes itself as a guided tour of Unix designed for a Mac user. Ah, just what the curiosity ordered.

For all my troubles, it was easier to get FreeBSD up and running (a mere two attempts instead of the several dozen with Fedora) and get a full graphical desktop running. To their credit, the Fedora Project is trying to create a “fully functional” graphical desktop environment right out of the box, which is a bit more of a challenge than FreeBSD’s standard install which drops the user straight into a command-line shell. Thankfully, I am not that much of a computer cripple, and so I was able to find my way into the full GNOME desktop environment in a matter of minutes.

Even so, there are many things left undone in FreeBSD’s standard installation (which are supposed to work almost immediately with a Fedora Linux installation), and which I will end up having to go back and configure myself later. For what its worth, I am at least getting a far more hands-on introduction to a real UNIX than what I would have gotten if Fedora Core worked as advertised. Whether or not that is a good thing only time will tell. In the mean time, I am merely happy to be experiencing a new OS.

Further Fedora Failures

I had tried to install Fedora Core 3 on several PCs of very different hardware makes last week. In each case I was unable to complete the Fedora Core 3 installer and get a working version of Fedora Linux onto the computers. Despite spending the better part of this week researching the topic and attempting to find solutions to the problems, I simply could not get it to work.

Fedora Core 3 Installation Problems

I was constantly confronted, in some point during either the graphical or text-based installation of Fedora with the console message Install exited abnormally followed by lines that read Sending termination signals…done., unmounting filesystems, and finally stating that You may safely reboot your system.

Whether or not it was, indeed, safe to reboot the system is up for debate. In some cases my previous OS had already been rendered unbootable by the installer having written a new partition table to the hard drive before “exitting abnormally.” This is why I had rescue CDs before attempting a Linux migration (and is why you need to have them, too)!

In all the mailing list archives and installation guides I could find, several common motifs echoed in the various “Help me” posts I perused:

1. Using certain kinds of mice, most notably any form of LogiTech® or serial mice can cause crashes of the graphical installer and should not be probed for automatically, or—in the worst cases—should not be used during installation, period. This is not really a problem because the installer provides ample keyboard navigation support, but is a rather annoying bug anyway.

   Workarounds that I have tried included:

   • Using a generic mouse instead of any major brand mouse. Specifically, I used an InterAct PS/2 (6-pin mini-DIN) mouse instead of a LogiTech, IntelliMouse, or serial mouse.
   • Unplugging the mouse for the duration of the automatic hardware probing and then manually configuring the installer to use a generic 3-button mouse.
   • Unplugging the mouse for the duration of the installer. Or for the durations that worked, anyway.
   • Using the text-based installer (which does not use a mouse at all) instead of the graphical one.

   Unfortunately, none of this had any effect other than to drive me crazy when trying to combine these various workarounds with the following workarounds.

2. Hardware probing itself is sometimes, allegedely, to blame for the failed installs. Disabling the hardware probe, or certain parts of it, by passing special options to the Linux kernel at boot time sometimes fixes the problem. Additionally, sometimes certain options regarding the power management features of the hardware and the kernel cause problems, and they should also be disabled during the installation boot-up process via special kernel arguments. The list of special boot options I tried were:

   • linux noprobe
   • linux noacpi
   • linux nofb
   • linux mem=mem_amount (with the appropriate amount of memory available on whatever system I was trying to install onto in place of mem_amount)
   • Various others I can’t remember right now.
   • Various combinations of the above options in conjunction with other attempts.

   In my case, the automatic hardware probe always managed to correctly guess my video card and monitor type (Intel 810 and ViewSonic EF70, respectively), so I was left to believe that the mouse was the only likely hardware issue. Nevertheless, I did notice that when the X server was starting up, parts of the usually gray screen were corrupted (think horizontal or vertical “snow” on an old TV set) and later, scrolling text and pop-up windows would not display correctly.

Sooner or later, usually sooner and almost always during the Searching for Fedora Core installations… progress dialogue, the X server would exit abnormally (or crash) and I would be told that it was safe to reboot my system. To which I scoffed and went to eat more cake.

I suspect that I may be lacking some kind of knowledge about either

• how the Linux kernel works or how OS kernels in general work, or
• how physical media such as hard disk drives interact with their software counterparts and how they store information (partitions, blocks, clusters, sectors, heads, cylinders, etcetera…)

that I need to understand in order to solve this problem. Unfortunately, despite all that I am reading on that topic these days I just could not get Fedora Core 3 to install successfully.

The really frustrating part in all of this is that there are no explicit error messages that come up to help me diagnose the issue. Even with the most braindead implementation issues on Windows, an error message, cryptic as it usually is, never fails to pop up and let the user know something is wrong. (Of course, the sheer frequency with which that happens is actually a prime motivator to my fondness for Unix-like systems.)

In this case, Fedora Core’s installer never warns me of a problem until its too late. In two cases, during a graphical installation on a machine that had already logged a failed installation attempt an error window did show itself and read something like BUG! Assertion (heads > 0) failed! which I understood to mean that something was either royally screwed with my partition table, BIOS drive geometry, or physical hard disk drive, and thus induced my search for the bugfix at LWN.net. Some common sense tells me it is more likely to be a problem with the partition table than anything else because both Windows and Knoppix work without any problems at all.

Fedora Core 2 Installation Succeses and Woes

It was suggested to me, when I finally gave up and sent a rare and detailed “help me” request to a very nice (and far more skilled) acquaintence of mine, that I should try to install Fedora Core 2 instead of Fedora Core 3 because it was likely to be far more stable.

The good news is that I have actually managed to get FC2 working on one (1!) machine of mine. Surprisingly, the docile computer is the extremely old PeoplePC-branded box with extremely low resources. (PeoplePC was a prevalent ISP in the mid-1990′s who were, I believe, the first to offer cheap computers as a bonus to new subscribers.)

Unfortunately, this machine runs painfully slowly (especially with a huge application like GNOME) and so even opening windows is an exercise in patience. Doing anything actually productive is simply unrealistic, as it took nearly 2 hours to get the latest updates in up2date from the Red Hat Network, and I actually cancelled the update in the middle of that process. So I had been hoping that Fedora Core 2 would install nicely on the other machines I have.

Of course, that’s when the bad news hit. The bad news is that bugs are in FC2 as well and the same thing happened with it as what happened with FC3 on one of the machines I tried to install it on. I was greeted with that very familiar Install exited abnormally message and was told to reboot my system. I spent a few minutes trying a few of the workarounds I talked about above on this version of Fedora on this machine, but gave up after about half an hour.

Interestingly, on the other machine which didn’t take the install the problem was entirely different. Rather than even getting to the graphical installer, when I pressed Enter at the boot:-prompt I was inexplicably dropped into the text-based installer instead. There was not the slightest indication that the computer was even trying to load the graphical installer. Wierd! This didn’t happen with Fedora Core 3 at all.

Optimistic as I was, I thought it still might work; perhaps it was just the graphical installer that wouldn’t function. But alas, this was not to be. When I was asked where the installation media is to be found and responded with “Local CD-ROM,” the text-based installer reported being unable to locate Fedora Core on the CD-ROM, even though the install disk was right there in the CD drive and I had used that very disc to get FC2 working on the first, oldest machine I tried!

Thus, I have decided that Fedora is a bipolar Linux distribution which needs heavy doses of appropriate patience and medications. I am at a total loss to explain this except to say that I am not left with a great taste for Fedora or Red Hat in my mouth. Ew.

FreeBSD is Less Moody, but also Potentially Confusing

To make matters even more confusing, I had tried to install FreeBSD some days ago on the old machine which now holds the only working Fedora Core installation I have. When I tried to install FreeBSD on it the other day I was unable to proceed due to an error that read something like Unable to locate Hard Disk Drive or similar. I can’t remember the exact wording.

After giving up with Fedora tonight, I tried to install FreeBSD on the machine which forced me into the text-based installer for FC2. It has completed without a hitch, and it only took me a few moments to orient myself within the system (man hier was incredibly helpful) in order to find out how to start a GNOME session and get back to the still-comforting GUI desktop.

Final Fumbling Thoughts

It might be of interest that both machines on which Fedora Core 2 failed to install for one reason or the other are HP Pavillion machines of somewhat different makes. One has a Pentium III processor and the other a Celeron, for instance.

Since FreeBSD has worked on one of these Pavillion towers but not on the old PeoplePC-branded tower (where Fedora 2 did work) I am hoping that at the very least some other BSD will take to the other Pavillion. That way, I can get different operating systems on my various computers and finally get to start playing with the many options out there, like I have been wanting to do for months now.

While I hope this is certainly not the end, I am not at all opposed to the idea of giving up on Linux (or at least Red Hat) for the time being. Perhaps all I really need is to go build myself a new system with new and (hopefully) supported hardware. That really shouldn’t be such a big deal, and I need to upgrade that old PeoplePC box anyway. 200-something mhz is just not acceptable these days. (It wasn’t even in the mid 1990′s, but PeoplePC was cheap and we are not rich.)

I am already far more comfortable with FreeBSD than I was even in the graphical environment of Fedora Core 2′s default GNOME themes. Perhaps it is the shell. I spend a lot of time on Mac OS X within Terminal at a tcsh prompt (though I have playing with bash lately) and, after all, FreeBSD is one of Mac OS X’s parents.

Seems only fitting that I should feel more comfortable with the BSD‘s than with a Linux. In the end, Linux Is Not UniX. ;)