Everything In Between

Let’s just jump into it.

Using Mac OS X, first make sure you have the git core installed as well as the git-svn components. Unlike other version control systems, git is not one monolithic program but a collection of smaller utilities, and the Subversion conduits are a subset of these utilities. The git-Subversion utilities themselves depend on having the Perl::Subversion bindings installed for Perl 5.

For the lazy MacPorts user, simply run:

sudo port install git-core +svn

This will install the git-core, the git-svn packages, as well as all the documentation for git and any required dependencies you don’t already have. Note that the documentation (man pages) is installed in /opt/local/man, which may not be in your default $MANPATH, so be sure to add that directory if man git returns a “No manual entry for git” error.

Alternatively, if you don’t want to use MacPorts, you can download a pre-compiled Mac OS X binary that includes git, the git docs, and the git-svn package from the git web site that comes complete with a standard GUI installation procedure.

Next, initialize a new git repository that tracks a remote Subversion one. This allows you to work privately using git, but to collaborate with other people who are using Subversion. Running

git svn init svn://my.svn.server/path/to/svn/repo workingcopy

will give you an empty git working copy named workingcopy configured to use the remote Subversion repository at svn://my.svn.server/path/to/svn/repo. This step is analogous to the need to run svnadmin create repo_db, to initialize a new repository database (where all the file versioning information will be stored). Unlike Subversion, git’s distributed database means that the working copy itself is also the location of the repository database, so there’s no need to deal with two filesystem paths anymore.

Next, change directory to your working copy, and run

cd workingcopy
git svn fetch

to populate your new, empty git working copy (and repository) with all the files from the remote Subversion repository.

Now that you have filled your git repository with a lot of data, if you want to, you can now also save a significant chunk of disk space by repacking that data into a more single, efficient, native packed git archive format (a .pack file). The git-repack -a command is used to do this, and its manual page says:

Instead of incrementally packing the unpacked objects, pack everything referenced into a single pack. Especially useful when packing a repository that is used for private development and there is no need to worry about people fetching via dumb protocols from it.

According to some sources, this can turn a 353MB Subversion repository into 31MB of git pack. Say:

git repack -a -d -f
Generating pack...
Counting objects: 284
Done counting 4475 objects.
Deltifying 4475 objects...
 100% (4475/4475) done
Writing 4475 objects...
 100% (4475/4475) done
Total 4475 (delta 1876), reused 0 (delta 0)
Pack pack-a115c320ff9c9968248bd250bdfea3110d0f0c1a created.
Removing unused objects 100%...
Done.

to perform the compression. For even greater savings, increase the compression by stipulating a high --window option, such as git repack -a -d -f --window=100. (--window defaults to 10—the higher the window, the harder git-repack will try to compress your stuff.) This turned a 36MB Subversion repository in one of my projects to a 20MB git pack. As always, your mileage may vary.

Congratulations, you have transformed your old Subversion repository to a git repository. All that’s left to do now is to get to the coding. Perhaps start by making a local (cheaper than cheap!) git branch….

That was easy, right? Most things in git are, in fact, that easy. If you’ve never used other version control systems before, be grateful. If you have, you can breathe a sigh of relief. Still not convinced? Don’t take my word for it…ask Linus Torvalds (or Randall Schwartz).

One of the pleasures of using Mac OS X is that Apple’s often-elegant UI makes the easy things easy and gracefully reveals the complexity of any complicated tasks you may wish to perform. Except when you want to do something out of the ordinary, of course, in which case things get complicated a bit more quicky. Thankfully, Apple doesn’t seem to go to any great lengths to cripple users, and this comes in handy time and time again.

While playing with my Desktop Backgrounds (computer “wallpapers”) the other day, I ran into an issue like this. I wanted to make sets of wallpaper images that overlapped one another. This is easily doable with Apple’s iPhoto, since all iPhoto albums appear as Desktop Background sets by default. I could simply put together a bunch of iPhoto Albums containing the images I wanted, including overlapping images, and be done with it. iPhoto takes care not to duplicate the photo into every album so space on my hard drive would not have been wasted.

But what if I didn’t want to use iPhoto? If I just have a bunch of image files on my disk in folders, how can I make overlapping sets? The solution is as simple as a Unix symbolic link—with a catch: only a symbolic link with an absolute path would work.

At first, I tried to simply use a standard Mac OS X “alias,” but that didn’t work. My MacBook Pro wouldn’t follow the alias file to the original file when trying to display the wallpaper. Then I tried a simple symlink from one file to another, but again it didn’t work.

Luckily, then I tried dragging the file into my terminal window, which automatically listed the full POSIX pathname of my image. This, surprisingly, worked. So it turns out that the Desktop & Screen Saver System Preference pane requires an absolute path to be specified if it’s to follow symlinks for creating wallpaper sets organized by folders. Go figure.

I have a suspicion that there are a lot of web developers out there who, like me, are eager to start learning more about Ruby but who are stunted by the incredible amount of unfamiliar conventions that are used in Rails. For many of us, our first introduction to Ruby was (or is) through Rails and the challenge of learning both Ruby and Rails at once should not be underestimated; frankly it’s damn hard.

Thankfully, there’s an easier way. Especially if you’re accustomed to “old-school” development workflows that you learned through, say, PHP or perhaps Perl scripting, then ditching Rails and starting purely with Ruby might be the way to go. Doing things on your own without the “magic” of Rails will let you learn Ruby in a simple, yet fully-featured environment, much like the way you might have learned classic CGI programming.

Specifically, by using eRuby, the Embedded Ruby interpreter, you can make Ruby web development feel just like PHP development. Install it, upload an HTML file with some embedded Ruby code in it, and—voila—you have a single-page web application.

Why this tutorial?

Hivemind’s Getting Started with Ruby on the Web is an excellent resource that described this process from start-to-finish for Windows users. In fact, it was after finding that article that I realized web development with Ruby didn’t have to be governed by Rails and, being a web developer like the authors of Hiveminds, I wanted an easy way to use my Ruby knowledge on the Web.

However, I couldn’t find any similar tutorial or walkthrough for those of us who use Macs. Even the Linux walkthroughs are lacking (or at least my Googling skill for them is). After spending a little while successfully getting eRuby set up on my Mac running Mac OS X 10.4 Tiger, I thought I’d share what I learned with others, so I wrote this article. Its content mirrors the one from Hiveminds to a great degree but focuses on getting eRuby installed on Mac OS X instead of Windows (obviously). I imagine these instructions should be pretty similar if not identical for any other *nix-like platform.

The Setup

Getting eRuby installed and configured with your web server is actually pretty straightforward and won’t take an experienced web developer much time at all. All we’re going to need is the Apache web server and the Ruby programming language to start, both of which are already pre-installed on Mac OS X 10.4 Tiger. (And in Mac OS X 10.5 Leopard, Apple is even shipping new Macs with Ruby On Rails preconfigured, but that’s a subject for another time.) However, Apple typically ships an older copy of Ruby than is readily available, so you might choose to use a newer version of Ruby that you download yourself.

The easiest way to get a copy of eRuby (and an updated Ruby, if you want it) is with a package manager such as MacPorts. With MacPorts installed (which is a simple, standard package installation downloadable from the MacPorts home page), simply run

sudo port install eruby

from your terminal to download the latest stable Ruby and eRuby in one fell swoop. After this runs, you’ll have a new Ruby in /opt/local/bin/ruby and an eRuby in /opt/local/bin/eruby. Alternatively, of course, you could compile and install eRuby yourself. Instructions for that are on the eRuby homepage.

As a quick test to see if everything is installed and working correctly, create a plain text file with your favorite editor that contains the following code:

Hello world! The time is now <%= Time.now %>.

All this does is print out a “Hello world!” greeting and then announces the current time (as reported by your system clock, of course).

If you named the above file helloworld.rhtml then you can feed this file to the eruby interpreter as follows:

eruby helloworld.rhtml

Note that the file extension doesn’t actually matter. You could use .erb, if you like, or any other arbitrary file extension. The file extension is, as you’ll see later, only used for telling Apache which kind of file it is. So once you come up with file extension you like, stick with it. The standard file extensions are either .rhtml or .erb, so I’d recommend using one of those. (I prefer .erb, myself, since I don’t necessarily only want to create HTML pages. I might want to create XML documents like news feeds or even XML databases, so .erb seems a more reasonable file name extension, though most of the eRuby documentation uses .rhtml.)

If things worked properly, you should see output similar to the following:

Hello world! The time is now Wed Oct 24 21:58:54 -0400 2007.

Serve it up!

Now that eRuby is installed and working, the next step is to set it up as a CGI so that when your web server (Apache, in my case) fetches the file it will first feed it through eRuby for processing before sending it back to the browser. The simplest way to do this is to configure Apache to use a specific directory somewhere on your filesystem as a CGI directory—that is, a directory whose contents are all treated as CGI programs. You do this via Apache’s ScriptAlias directive.

Conveniently, standard installations of Mac OS X already come with a CGI directory. It’s located at /Library/WebServer/CGI-Executables. This directory is just like any other, except that Apache treats it specially. It treats it specially because Apache’s configuration file, /etc/httpd/httpd.conf contains a line of text that reads as follows. (On a standard installation of Mac OS X client, this is line 671 of the file.)

ScriptAlias /cgi-bin/ "/Library/WebServer/CGI-Executables/"

This line basically just tells apache, “Treat every file in the /Library/WebServer/CGI-Executables directory as though it were a CGI program.” So, since that’s how we’re treating eRuby, we’re going to want to put our eruby binary in that directory. However, we don’t actually want to make copies of the binary. Having too many copies of a program on your system can easily get confusing. So instead, we’ll make a symbolic link (or an alias to use the classic Mac terminology) to the binary.

ln -s /opt/local/bin/eruby /Library/WebServer/CGI-Executables/eruby

With our eruby CGI executable in the proper place for Apache to find it, we can now tell Apache which files we want it to send to this program. We do this using the AddHandler and Action directives.

AddHandler simply tells Apache to treat files with a certain extension in a certain way. For instance, you can say something like, “Treat all files ending in .php as PHP scripts.” Similarly, you can (and will) tell Apache, “Treat all files ending in .rhtml or .erb as eRuby CGI pages.”

To do this, we add the following line to our /etc/httpd/users/username.conf file somewhere inside the <Directory> and </Directory> block. Naturally, replace username with, of course, your Mac OS X user’s short name:

AddHandler rubypage .erb .rhtml

Again, this just tells Apache that files ending in .erb and .rhtml should be treated the same way other rubypages are treated. But how are eRuby pages supposed to be treated? They’re supposed to be handed off to the eRuby program, of course, so we need an Action directive that applies to the rubypage handler we just defined.

Right below the above line, add the following line, too:

Action rubypage /cgi-bin/eruby

This line tells Apache that any file that it is treating as a “rubypage” file should be sent to the program at the URL accessible at /cgi-bin/eruby, which we’ve defined earlier to be our symbolic link to the eruby interpreter.

Your completed user-specific Apache configuration file should now look something like this:

<Directory "/Users/username/Sites/">
…
    # Let eRuby files get parsed through the proper CGI binary
    AddHandler rubypage .erb .rhtml
    Action rubypage /cgi-bin/eruby
</Directory>

Are You Being Served?

That’s it, everything should now be in place. To make sure, move your earlier helloworld.rhtml file into your user’s Sites folder, and now simply point your browser to http://localhost/~username/helloworld.rhtml (again, replacing username with your Mac OS X user short name, of course). Congratulations, you’re serving up Ruby-coded applications via eRuby.

Additional Reading

This is just the set up, of course. Now the real fun begins: learning Ruby. Here are some suggested places to start.

• Programming Ruby: Ruby and the Web — A chapter out of the free book Programming Ruby all about Ruby as a CGI programming language.
• Hiveminds: eRuby: How to use Ruby CGI and ERB templating for Web Pages — a quick’n'dirty article explaining the basics of using Ruby on web pages in classic CGI style where program logic and HTML templates are kept separated, as opposed to classic PHP style, where the two are mingling together on one page.
• COOLNAMEHERE: Simple Ruby CGI — Like the Hiveminds example above, another quick’n'dirty Ruby CGI how-to.
• cgi: Ruby Standard Library Documentation — Ruby’s standard library contains a rich set of features, and the CGI objects are among them. They are well-documented in the official Ruby Documentation Project.

Naturally, feel free to leave comments if you have more resources that you found helpful.

This is already horribly old news, and by old I mean several days ago since that’s about as fast as it takes technology news to grow old, but Apple is releasing Mac OS X 10.5 “Leopard” at the end of this month. Apple is calling this release a “major upgrade,” and indeed Apple has rarely made its users wait so long between operating system releases as they have done between Tiger (Mac OS X 10.4) and Leopard. So, I’m already excited.

But then today I was glossing over Apple’s featured features list and I got even more excited. There are the usual, largely meaningless, fluff updates that are nice for Joe Schmo or his mother, but that power users simply don’t care about, like the new iChat support for animated buddy icons, but the list is also chock-full of really cool, really useful features.

What’s interesting is that a good deal of these features aren’t really new features at all. For instance, if you knew how to manipulate the NetInfo database on your Mac, you could already share any folder via Apple’s “Personal File Sharing” feature. (Here’s a Mac OS X Hints hint explaining how to do it.) In Leopard, however, Apple claims that this functionality is now integrated straight into a folder’s Get Info… window. If it works as smoothly as Apple claims, this is finally going to bring Mac OS X (client) into decent competition with Windows XP Professional in terms of GUI-level power-user features.

However, while all of these features are really cool, here’s a list of the ten geekiest features I will probably absolutely love, for one reason or another.

• Ruby on Rails, out of the box — The hot thing in web development right now is Ruby on Rails. Macs have already been the best personal desktop and web development platform because they have built-in support for the Apache web server and a host of other features, but now they will come with a ready-to-roll installation of Ruby on Rails, sporting Mongrel and (better yet) Capistrano! Specifically with the addition of Capistrano, which is terribly undersold as simply a Ruby on Rails deployment platform, these UNIX-y “toolbox” items are bound to make Macs that much more useful right out of the box.
• Safari’s full history search — As their recent public partnerships with Google have shown, Apple is very clearly invested in search technologies. Spotlight gets a huge number of improvements in Leopard, but none which I think are going to be more useful to more people than this one: spotlight searches on the full text of each web page in your visited history list. That’s just awesome. Also awesome: using spotlight as a calculator and as a dictionary, which also shows just how Google-like Apple is trying to be. (Google also lets you ask it arithmetic questions and a dictionary.)
• Wikipedia articles in Dictionary.app — I love Wikipedia because it’s one of the fastest ways to get (relatively) reliable information quickly. Now that Dictionary.app has built-in integration with Wikipedia, imagine the possibilities for getting that knowledge instant-gratification craving fixed. Apple has not yet announced this capability, but I can easily envision a scenario where all Cocoa text fields are instantly “wikified” (with text that matches Wikipedia articles highlighted) much in the same way that current Cocoa text fields allow you to right-click on a misspelled word and have it corrected by Dictionary.app.
• Application-based firewall — In classic Apple fashion, functionality that was previously available via third-party additions is now available from Apple itself. In this case, I have to wonder how well Apple’s updates to its firewall will obviate the need for Little Snitch, which is basically an application-based firewall, too, and a good one at that.
• Built-in guest log-in account — If you’re as paranoid about security as I am, you’ve already created a special, limited-access user on your system (called Guest or Visitor or whatever) and whenever friends are over, you tell them to use that account instead of your own. Now in Leopard, Apple has gone through the trouble of setting this up for us already. A small change that is going to have a big impact.
• Scriptable System Preferences & applications — With AppleScript, you can automate the things your computer does with scripts, as long as those things are “scriptable.” In previous versions of Mac OS X, huge gaping holes of what things shipped by Apple were scriptable existed, causing me (personally) some really annoying headaches. AppleScript GUI scripting helped me get around many of those roadblocks, but now it seems Apple is finally filling in some of the most notorious gaps in this functionality with scriptable System Preferences. Yay!
• Automator workflow variables — Automator brings the power of AppleScript I just mentioned to more people with a completely graphic programming environment. There is no need to open up a text document and write AppleScript code because Automator lets you create a script (called a Workflow in Automator jargon) using your mouse by dragging and dropping actions into the order you want them to be performed. It’s very slick, but until now it’s been very limited. With Leopard, Apple is beefing up Automator so that it includes things like variables, basic programmatic capability that was sorely lacking before. (Also majorly cool: a command-line utility to access Automator!)
• Finder.app’s path bar — Every serious Mac user knows that the Finder needs a lot of help. Now, it’s getting some. Something the Windows Explorer has had forever (as had every desktop environment for Linux, of course) is a visual cue to show you where in your filesystem tree a given folder is located when you are viewing said folder. Now the Finder gains this capability (though Apple’s description implies that it’s going to be off by default) with what Apple is calling a “Path Bar”. Finally!
• Cocoa and scripting bridges — Even though no one really seems to know about it, it has long been possible for languages other than AppleScript to do things like send Apple Events to Mac OS X applications. Specifically, Ruby and JavaScript, two of the most well-known web development languages in existence, can already do this with a single ScriptingAddition (OSAX). But now Apple is making this functionality a central feature and fully extending it to their Objective-C (and Cocoa) language and applications such as Xcode and Interface Builder. This means people like me will have a shallower learning curve before we’re able to create full-fledged, native Mac OS X applications. Now that’s exciting!
• Xcode 3 refactoring — This is something you kind of have to see to believe. I got the opportunity to see it demoed at Apple’s Leopard Tech Talks last year and I was really excited by it. With the new Xcode, Apple’s development IDE, you can do away with find-and-replace searches for things like renaming functions because Xcode understands what parts of your code are what structures and, when you tell it to “change the function named myFunction to myNewFunction,” it’ll only find-and-replace function names instead of every instance of the string “myFunction.” That’s pretty big, and if it were available for more languages, it’s almost enough to make me ditch vim.

So there you have it. Ten features you might not have already known about that are some of the most promising features I can see in Leopard. And I didn’t even get into Wide-Area Bonjour, which could make services like DynDNS or No-IP a thing of the past (and which I still want to learn more about), or the new Terminal application (finally with tabs!), or even the multiple user certificates for S/MIME encrypted email.

Note: One of the least known security features available on Mac OS X is also possibly one of the best, and the simplest. Evidently, all Intel-based Macs are shipped with the XD (aka. NX, aka. DEP) bit turned on—and thankfully there doesn’t seem to be any way for users to turn it off. However, this isn’t a silver bullet and if you want to learn why you should check out this excellent Anandtech article: A Bit About the NX Bit.

I recently purchased a new iMac. It’s aluminum. And glass. And shiny. And it’s so much faster than my old G4 workstation I feel a little bit like I’ve just come out of the stone age.

Anyway, the experience of moving one’s data from one machine to another is always a bit of a hassle. There are so many little things you’re sure you’ve forgotten, preferences you don’t want to have to recreate, and small gotchas that, if you’re not careful, will mean you’ve lost something like your old email or your contacts—sometimes for good.

Thankfully, with a remarkably few easy planning steps, a lot of this trouble can be avoided all together. Much of this is thanks to the brilliant architecture of Mac OS X, which does a superb (though not perfect) job of separating data from applications. The other major benefactor of this kind of easy transition is server-based data stores, or hosted services (even if those services are ones I’ve hosted myself).

So here’s what I wanted to move over from my old mac to my new one:

• Personal documents, pictures, movies, and music, etc.
• Preferences for all applications, such as my dock layout, the Finder’s and Mail’s toolbar button arrangements, and so on.
• PIM databases, such as my Address Book contacts, iCal calendars, my email, and the like.

This could really be a much longer list, especially if I were going to name every single item of personal importance. Nevertheless, each item falls into one of these three main categories of stuff that I want to move.

The first time you turn on a new Mac it will ask if you want to transfer your data from your old computer to your new computer. It does this using the Migration Assistant application located in your Mac’s /Applications/Utilities folder. While this works well for the most part, I’ve seen this program fail one too many times—especially for cross-architecture, PPC to Intel, transfers—that I’m simply not a fan of using it. Besides, as you’ll see in a moment, it’s trivially easy to do all of that yourself.

Of course, all of your personal information should be somewhere in your Home folder. You could just copy the entire directory from one Mac to the other, but you’ll probably end up deleting some (albeit probably non-critical) items from your new machine. When you drag-and-drop in the Finder, the Finder first removes anything of the same name on the destination as the source of the transfer. This is not what you want.

So here’s how I dealt with moving each type of item, after ensuring I had a good backup:

Transferring personal documents

rsync is a godsend for any situation where you want to ensure that you’re copying files from one location to another. It compares the contents of folders and, by default, only adds missing items to the destination while leaving everything else alone. It also overwrites identically-named files on the destination with files from the source.

To use rsync, I first needed to turn on the destination Mac’s “Remote Login” feature; this turns on the SSH server. (Find this in System Preferences → Sharing → Services.) Next, simply issue this relatively simple command at the command line, from your home directory:

rsync -aEve ssh Documents UserShortName@DestinationMac:Documents

This tells rsync to copy files from the local machine’s Documents directory to the DestinationMac‘s Documents directory in archive mode (the -a option), which preserves numerous file meta-data such as its modification times and access permissions, along with the file’s extended attributes and resource forks, if any exist (-E), reporting verbosely (-v) as it does so, executing over SSH (-e ssh). The UserShortName is the DestinationMac‘s user short name, obviously, and the DestinationMac is either its hostname (such as mynewmacintosh.local) or its IP address (such as 192.168.0.2).

I repeated the above step for the Desktop, Movies, Music, Pictures, Public, and Sites folders. Most of my music is actually on an external hard drive, so that was of course a cinch, and rsync took care of the rest.

Transferring preferences

Mac OS X stores its user-specific preferences in each user’s home Library/Preferences folder. The set up is elegant and beautifully implemented. Just like the folders above, I could have simply rsync‘ed my preferences over, but I wanted to be more selective. So I simply scrolled through the list finding the preference files for the applications I still use and copied them over. I used the Finder’s drag-and-drop, but you could use whatever method you want.

These files are named in reverse-DNS style notation, so for instance, my Mail preferences were stored in the file com.apple.mail.plist.

This was a super-simple step.

Transferring PIM databases

Like preferences, user-specific data is often stored in that user’s home Library/Application Support folder. Inside that folder is a folder for each application that has some data to store. All of iCal’s information, for example, is thus stored in your home Library/Application Support/iCal folder.

rsync to the rescue again:

rsync -aEve ssh Library/Application\ Support/iCal "UserShortName@DestinationMac:Library/Application\ Support/iCal"

In this case, because of the space in the Application Support folder’s name, the quotations and the backslashes are required to help rsync find the proper folders. Other than that, the command is the same as before.

I selectively copied over all the data from the applications I wanted to keep and, voila, the next time I opened iCal or Address Book, my data was there.

The Mail application and Safari both have special ways of handling their information, however, so their datastores couldn’t be found in the Application Support folder. Instead, they keep their folders directly inside the Library folder named (predictably) Mail and Safari, respectively.

Also, since I use IMAP for email and shared calendars in the form of .ics subscriptions whenever possible, the natural sync process for these databases saved a lot of time.

Most other programs in non-standard locations could be found with a little digging inside my Library folder.

It should be noted that for this to work reliably, you should always move all the preference files along with the Application Support directory for any program whose Application Support directory you’re grabbing!

Cleaning up

All told, the whole thing took maybe two hours and most of that time was spent transferring data. It was really quite painless. Because this was a surgical transfer, instead of a full “migration,” there’s very little to clean up.

However, for some reason, Apple’s .Mac Sync service had quite a bit of trouble keeping up with my new Mac’s new additions. The fix was simple enough: unregister every sync’ed computer, then reset all of the .Mac Sync data with the data on the new Mac. Then, re-sync each device either merging its data during its syncing process or replacing the data on it with the new, known-good data on .Mac. (Again, make sure your backup works before you do this.)

And that’s pretty much it. Now I have a new iMac, but it looks exactly like my old one on the screen. :)

More and more often I find myself doing my work on the command line because it’s faster and more efficient. However, most users find that they end up spending a lot of time typing and re-typing commands, editting them, and then having to fix them. This is especially true if you’re new to the command line.

In addition, many times you want to run multiple commands with only slight variations. Perhaps you are benchmarking your Web server with ApacheBench and want to change some of its settings. Whatever you are trying to do, many shells actually offer a full suite of command-line editting shortcuts.

Some of them are built into the shell itself and others are configured elsewhere. Unfortunately, different distributions and different operating systems sometimes don’t have the same shortcut configuration. There are lots of lists out there for Bash keyboard shortcuts for Linux, and many of these work on Mac OS X, but they don’t all work.

So here’s a list as complete as I can make it of the default keyboard editting shortcuts configured on a Mac OS X machine running bash. An extra goodie is that some of these movement shortcuts work in any Coacoa text field, which means you don’t have to keep switching between bash’s command line movement shortcuts and the +arrow keys shortcuts anymore.

CTRL+a

Move cursor to beginning of the line. (I think of this as anchoring my cursor to the start.)

CTRL+e

Move cursor to the end of the line.

CTRL+k

Delete everything from under the cursor to the end of the line. (I think of this as killing the rest of my line.)

CTRL+f

Move forward one character. Identical to →.

CTRL+b

Move backward one character. Identical to ←.

ESC+f

Move forward one word.

ESC+b

Move backward one word.

CTRL+u

Delete everything from under the cursor the beginning of the line.

CTRL+w

Delete from under the cursor to the beginning of the word.

CTRL+r

Recall previous commands by searching for them.

CTRL+t

Transpose (swap) the two characters before the cursor with one another.

ESC+t

Transpose (swap) the two words before the cursor with one another.

CTRL+y

Paste the most previously-deleted string. Basically a sort of command-line editting “undo.”

CTRL+z

Stop the current process and send it to the background.

CTRL+c

Send an SIG_HUP to the current process. The net effect of doing this on the command line is that you cancel your current command and are presented with a blank new line.

CTRL+d

Send an end-of-file special character to the current process. Doing this at the command line is identical to closing your terminal window.

CTRL+p

Recall previous command executed. Identical to ↑.

CTRL+d

Forward delete.

CTRL+h

Backspace.

CTRL+j

Carriage return. Identical to hitting the return key.

CTRL+m

Newline. Identical to return.

CTRL+l

Repaint screen. This is useful if a program’s output is overwriting some text on your terminal. The effect of doing this on a command line is that you clear the screen. Note than in Apple’s Terminal.app, you can also press +K to clear the screen.

CTRL-x + CTRL-x

Mark current location in line and jump to beginning of line or second mark if defined. Repeat to jump to between both marks.

CTRL+v

Insert next character verbatim. This is how you escape control sequences. For instance, to literally send a

CTRL+[

Identical to ESC.

ESC+c

Capitolize word under cursor and move to next word.

ESC+u

Uppercase word under cursor and move to next word.

ESC+l

Lowercase word under cursor and move to next word.

ESC+.

Insert last word from previous command after cursor.

TAB

Auto-completes file, folder, and program names.

ESC-?

List the possible completions

CTRL-x /

List the possible filename completions

ESC-/

Attempt filename completion

CTRL-x ~

List the possible variable completions

ESC- ~

Attempt username completion

CTRL-x $

List the possible variable completions

ESC-$

Attempt variable completion

CTRL-x @

List the possible hostname completion

ESC-@

Attempt hostname completion

CTRL-x !

List the possible command completions

ESC-!

Attempt command completion

ESC-TAB

Attempt completion from previous commands in the history list

Note that this list assumes you’ve not reconfigured any of these shortcuts. Again, they’re the default keyboard assignments. Furthermore, they’re the default keyboard assignments for the default command-line editting mode, which is emacs mode. If you’re familiar with the vi editor, you can change this to use all the cursor movement and deletion keys you’re familiar with by saying set -o vi to your shell.

External references

Here’s some other people’s lists. They’re basically the same, but what the hey.

• BASH Command-Line Editing at lxpages – Also has a listing of vi-mode key shortcuts.
• The BASH Reference Manual for Command Line Editing
• Bash Features – Command Line Editing – discusses even more commands that Readline can do for you when using bash.

In a previous post I wrote a small expect script to update a remote web server’s deployed code on a new commit to a Subversion repository using Expect and Subversion’s post-commit hooks. That first script was extraordinarily basic, so I’ve been wanting to add some sanity and error checking to it for a while. I finally got around to it today.

This improved version of the post-commit hook does the same thing as the last one (that is, it logs into your web server over SSH with the given user and password, and yes, I’m aware of the scariness of embedding a password in such a way, so you should really set up SSH to use public keys for authentication for this), except now it also produces useful output.

Here’s the same script as before, but improved:

#!/usr/bin/expect -f

#
# AUTHOR: Meitar Moscovitz 
# DATE  : Thu Jun 21 16:32:42 EDT 2007
#

set HOST my.web.server
set USER someuser
set PASS xxx

# the working copy we're going to update
set WC /path/to/working/copy

# the path to the svn executable on the remote web server
set SVNBIN /usr/local/bin/svn

# our network is slow, set a long timeout
set timeout 30

##### DO NOT EDIT PAST THIS LINE! #####

# POST-COMMIT HOOK
#
# The post-commit hook is invoked after a commit.  Subversion runs
# this hook by invoking a program (script, executable, binary, etc.)
# named 'post-commit' (this file) with the
# following ordered arguments:
#
#   [1] REPOS-PATH   (the path to this repository)
#   [2] REV          (the number of the revision just committed)
#
# Note that Subversion does not provide this program with an environment
# of any kind. That means this program lacks a current working directory,
# a home directory, a $PATH, and so on.

set REPOS [lindex $argv 0]
set REV [lindex $argv 1]

# Define error codes
set E_NO_SSH     1 ;# can't find a usable SSH on our system
set E_NO_CONNECT 2 ;# failure to connect to remote server (timed out)
set E_WRONG_PASS 3 ;# password provided does not work
set E_UNKNOWN   25 ;# unexpected failure

# find the SSH binary on our system
if {[file executable /usr/bin/ssh]} {
	set SSHBIN /usr/bin/ssh
} elseif {[file executable /usr/local/bin/ssh]} {
	set SSHBIN /usr/local/bin/ssh
} else {
	send_error "Can't find a usable SSH on this system.\n"
	exit $E_NO_SSH
}

spawn $SSHBIN $USER@$HOST $SVNBIN update $WC

expect {
    "continue connecting (yes/no)? " { send "yes\r"; exp_continue; }
    -nocase "password:" { send "$PASS\r"; }
    timeout {
        send_error "\nWe have timed out after $timeout seconds while trying to connect to $HOST!\n";
        exit $E_NO_CONNECT;
    }
}

expect {
	-nocase "password:" { ;# if we are asked for the password again, then we have provided the wrong password
		send_error "\nCan not log in to $HOST because the password provided for user $USER has been rejected.\n";
		exit $E_WRONG_PASS;
	}
	-re "revision (\[0-9]+)." {
		if {$REV == $expect_out(1,string)} {
			send_user "\nSuccessfully updated $WC on $HOST to revision $REV.\n"
		} else {
			send_user "\nUpdated repository to revision $expect_out(1,string), but svn reports that we are at revision number $REV.\n"
			send_error "CAUTION: Repository updated to revision $expect_out(1,string), but committed revision $REV.\n"
		}
	}
	default {
		send_error "An unexpected error has occured. The process at spawn ID $spawn_id has produced the following output:\n"
		send_error $expect_out(buffer)
		exit $E_UNKNOWN
	}
}

In one of my web development projects, it became important to keep the staging web server in sync with the latest code that myself and several other developers were working on. There are a number of ways to mirror files and directories across machines, rsync being one of the most widely known. However, in addition to simply mirroring the files across the two servers, we also needed a way to kick off the mirroring process that cleanly integrated with our development workflow. Subversion’s post-commit hook allowed us to do just that.

Still, however, the problem was not exactly straightforward. We needed to execute a svn update command on a server other than the server on which the Subversion repository was being hosted. Shell scripts are the obvious choice for command-line automation in UNIX, but they don’t deal with interactive commands very well. So instead of writing a shell script, I wrote an expect script.

This expect script is really basic. There’s a better one in a future post on this topic.

#!/usr/bin/expect -f

#
# This expect post-commit hook connects to staging-webserver
# and updates the working copy hosted there to the latest checked-in code.
# This means that whenever code is committed to the repository, the web site hosted
# will always be running the latest version of the code.
#
# AUTHOR: Meitar Moscovitz
#

# POST-COMMIT HOOK
#
# [...]
#

set REPOS [lindex $argv 0]
set REV [lindex $argv 1]
set HOST staging-webserver
# Use update-user to log in to staging-webserver
# to update the working copy, but this can probably be improved so as not
# to expose this user's password.
set USER update-user
set PASS update-user's-password

spawn /usr/bin/ssh $USER@$HOST svn update /path/to/web/site/directory
expect "continue connecting (yes/no)? "
send "yes\r"
expect "password: "
send "$PASS\r"
expect eof

There’s one really tricky bit to this script, which is the understanding that when Subversion runs the post-commit hook, no environment information is passed to this script. As a result, there is no home directory or path information set for this executable. This is why everything is defined using absolute paths. Also, because there is no home directory for update-user, the expect script will always be prompted by SSH to re-verify the server’s identity. So rather than just expecting “password: “, we always expect “continue connecting (yes/no)? ” and say yes, and then send our password.

Note, of course, that update-user should be a user with limited access to the system, yet enough so that he may update the working copy on the web server. I’m sure there is probably a more secure way of doing this as well, so any sort of feedback on securing this or scaling it would be welcome.

I’ve always loved teaching (for many reasons, some of which are too arrogant to get away with mentioning) and today my supervisor asked me if we could spend a little while talking about the command line and Mac OS X’s UNIX underbelly. Of course, I was thrilled at the prospect (see earlier statement on my love of teaching) and so I offered up a couple of resources. Since I think they’d be helpful for a lot of Mac users who’ve gotten bit by the UNIX bug (no pun intended), I expanded my list into a mini-guide.

Shells and Terminals: What’s the difference?

The very first thing anyone thinks about when they think UNIX is the command line interface (or CLI, for short). The funny thing about the CLI is that it’s actually a program in and of itself called a shell that takes what you write on the command line and basically translates it to the computer so it will do what you tell it to do.

So the computer (or the OS) is really only accessible through this shell. This is fairly straightforward until you realize that the shell is usually only accessible through another program, commonly called a terminal, which on Mac OS X is called Terminal.app. Therefor the terminal is really a window to a shell which is another window to the OS, which then talks to the computer for you. So when people say they want to learn to use the command line, you can understand the confusion that might cause. What shell do they want to use? And on top of that, which program are they intending to use to access their shell? Even on the Mac, there are at least two built-in terminals you can use (Terminal.app and XTerm) and maybe half a dozen shells.

As far as terminals go, Terminal.app is the most accessible (and you can find it in your Utilities folder in your Applications folder on your startup disk). My favorite resource for getting started with Terminal was MacDevCenter‘s excellent article, Learning the Mac OS X Terminal, the title of which makes a lot more sense after understanding the previous two paragraphs. That article will show off some of Terminal’s impressive features that make a GUI environment complement the command line environment nicely.

The Command Line: Speak and it will be so

The command line itself is, of course, yet another subject entirely. Thanks to its spartan appearance, I think the best way to learn it is to just use it. If you’re feeling brave and have a spare computer lying around, take the leap and install a Linux or a BSD distro, but run it all from the command line (which is simply a matter of changing the active runlevel on most UNIX-like OS distros). This will effectively boot you into a shell, with no GUI coating. For Mac users who want to stay closer to home, you can do the same thing by logging into Mac OS X as the >console user (with no password). This strips Mac OS X of its Aqua interface and drops you into a login shell.

The important thing to remember is that, while in a shell, you can do everything you can do in a GUI mode. You can browse the web, edit and view documents, and even read and compose email. Of course, there won’t be any pictures, but the point remains that these things can be done. A week of running your computer in a shell and you’ll feel more comfortable than ever at the command line.

But while you’re there, you’ll want to acquaint yourself with several commands you can use to get help. The most important is the man command, which will let you browse the documentation for any command you want to use. You can even browse the documentation on the man command itself by typing man man (and I strongly recommend that you do that).

Another command you’ll want to get familiar with is apropos. If you type man apropos, you’ll get a very description of what that command does. (I won’t spoil the treat for you.) A similar command to apropos is whatis. All these commands are just programs like any other, absolutely identical to the ones you’re already using with the sole exception that instead of outputting pixels to a screen, they output characters to a shell.

On the web, one of my oft-referenced resources for learning the basics of a shell environment (and thus the command line) is TuxFiles. It’s geared towards Linux users, but the things you can learn there apply to all UNIX-like operating systems. Another one is LinuxCommand. Check them both out.

Inside the Shells

By now you’re hopefully running your computer with a terminal window constantly open in the background, or at least you’re excited about trying it. So what’s next? Learn a new shell! By default, most OS‘s will use bash, an abbreviation for the Bourne Again Shell, which itself is based on the Bourne Shell (get it, get it?), written by Steve Bourne.

Different shells all do the same thing, which is enable you to speak with (by typing to) your computer and have it do your bidding, but they understand different dialects so you need to speak to them in slightly different ways. There are dozens of shells you can use, and which one you ultimately choose depends on the features and syntax of each. I spent a good deal of time reading about them all on Wikipedia before I decided to scrap the whole ordeal and stick with Mac OS X’s default bash shell. However, I learned a lot along the way, and that was really the point.

Speaking Command Line is like learning a new language

Ultimately however, it’s not about the shell or terminal you use, but about the things you can do. This, unfortunately, has to come with time. It’s exactly like learning a language. Every command you learn is akin to learning and using a new verb. The more you use a new language to communicate, the better you’ll become at speaking it. The same is true of the command line.

Keep your terminal program open 24/7/365, and whenever you’re doing something, ask yourself if you could do the same thing on the command line. If you’re not sure, Google it, and if the answer is yes, do it that way the next time. This is especially useful when what you’re doing is more than one thing. It takes a dozen or so clicks to open a file in TextEdit, copy it to the clipboard, past it to a new document, then add some text to the end of the new document, and finally save it again, but all of that can be done by typing a single line in a terminal. (Hint: use your shell’s piping and I/O redirection features.)

The more you immerse yourself, the better you’ll become. If you’re really dedicated, I’ve heard nothing but great things about the book Learning Unix for Mac OS X Tiger, which describes itself as a guided tour of Unix designed for a Mac user. Ah, just what the curiosity ordered.

Fedora Core is a bipolar Linux distribution. There, I’ve said it. I know there are legions of Fedora devotees out there, but they have probably had no such headaches installing the damn distro as I have had in the past week or so.

For all my troubles, it was easier to get FreeBSD up and running (a mere two attempts instead of the several dozen with Fedora) and get a full graphical desktop running. To their credit, the Fedora Project is trying to create a “fully functional” graphical desktop environment right out of the box, which is a bit more of a challenge than FreeBSD’s standard install which drops the user straight into a command-line shell. Thankfully, I am not that much of a computer cripple, and so I was able to find my way into the full GNOME desktop environment in a matter of minutes.

Even so, there are many things left undone in FreeBSD’s standard installation (which are supposed to work almost immediately with a Fedora Linux installation), and which I will end up having to go back and configure myself later. For what its worth, I am at least getting a far more hands-on introduction to a real UNIX than what I would have gotten if Fedora Core worked as advertised. Whether or not that is a good thing only time will tell. In the mean time, I am merely happy to be experiencing a new OS.

Further Fedora Failures

I had tried to install Fedora Core 3 on several PCs of very different hardware makes last week. In each case I was unable to complete the Fedora Core 3 installer and get a working version of Fedora Linux onto the computers. Despite spending the better part of this week researching the topic and attempting to find solutions to the problems, I simply could not get it to work.

Fedora Core 3 Installation Problems

I was constantly confronted, in some point during either the graphical or text-based installation of Fedora with the console message Install exited abnormally followed by lines that read Sending termination signals…done., unmounting filesystems, and finally stating that You may safely reboot your system.

Whether or not it was, indeed, safe to reboot the system is up for debate. In some cases my previous OS had already been rendered unbootable by the installer having written a new partition table to the hard drive before “exitting abnormally.” This is why I had rescue CDs before attempting a Linux migration (and is why you need to have them, too)!

In all the mailing list archives and installation guides I could find, several common motifs echoed in the various “Help me” posts I perused:

1. Using certain kinds of mice, most notably any form of LogiTech® or serial mice can cause crashes of the graphical installer and should not be probed for automatically, or—in the worst cases—should not be used during installation, period. This is not really a problem because the installer provides ample keyboard navigation support, but is a rather annoying bug anyway.

   Workarounds that I have tried included:

   • Using a generic mouse instead of any major brand mouse. Specifically, I used an InterAct PS/2 (6-pin mini-DIN) mouse instead of a LogiTech, IntelliMouse, or serial mouse.
   • Unplugging the mouse for the duration of the automatic hardware probing and then manually configuring the installer to use a generic 3-button mouse.
   • Unplugging the mouse for the duration of the installer. Or for the durations that worked, anyway.
   • Using the text-based installer (which does not use a mouse at all) instead of the graphical one.

   Unfortunately, none of this had any effect other than to drive me crazy when trying to combine these various workarounds with the following workarounds.

2. Hardware probing itself is sometimes, allegedely, to blame for the failed installs. Disabling the hardware probe, or certain parts of it, by passing special options to the Linux kernel at boot time sometimes fixes the problem. Additionally, sometimes certain options regarding the power management features of the hardware and the kernel cause problems, and they should also be disabled during the installation boot-up process via special kernel arguments. The list of special boot options I tried were:

   • linux noprobe
   • linux noacpi
   • linux nofb
   • linux mem=mem_amount (with the appropriate amount of memory available on whatever system I was trying to install onto in place of mem_amount)
   • Various others I can’t remember right now.
   • Various combinations of the above options in conjunction with other attempts.

   In my case, the automatic hardware probe always managed to correctly guess my video card and monitor type (Intel 810 and ViewSonic EF70, respectively), so I was left to believe that the mouse was the only likely hardware issue. Nevertheless, I did notice that when the X server was starting up, parts of the usually gray screen were corrupted (think horizontal or vertical “snow” on an old TV set) and later, scrolling text and pop-up windows would not display correctly.

Sooner or later, usually sooner and almost always during the Searching for Fedora Core installations… progress dialogue, the X server would exit abnormally (or crash) and I would be told that it was safe to reboot my system. To which I scoffed and went to eat more cake.

I suspect that I may be lacking some kind of knowledge about either

• how the Linux kernel works or how OS kernels in general work, or
• how physical media such as hard disk drives interact with their software counterparts and how they store information (partitions, blocks, clusters, sectors, heads, cylinders, etcetera…)

that I need to understand in order to solve this problem. Unfortunately, despite all that I am reading on that topic these days I just could not get Fedora Core 3 to install successfully.

The really frustrating part in all of this is that there are no explicit error messages that come up to help me diagnose the issue. Even with the most braindead implementation issues on Windows, an error message, cryptic as it usually is, never fails to pop up and let the user know something is wrong. (Of course, the sheer frequency with which that happens is actually a prime motivator to my fondness for Unix-like systems.)

In this case, Fedora Core’s installer never warns me of a problem until its too late. In two cases, during a graphical installation on a machine that had already logged a failed installation attempt an error window did show itself and read something like BUG! Assertion (heads > 0) failed! which I understood to mean that something was either royally screwed with my partition table, BIOS drive geometry, or physical hard disk drive, and thus induced my search for the bugfix at LWN.net. Some common sense tells me it is more likely to be a problem with the partition table than anything else because both Windows and Knoppix work without any problems at all.

Fedora Core 2 Installation Succeses and Woes

It was suggested to me, when I finally gave up and sent a rare and detailed “help me” request to a very nice (and far more skilled) acquaintence of mine, that I should try to install Fedora Core 2 instead of Fedora Core 3 because it was likely to be far more stable.

The good news is that I have actually managed to get FC2 working on one (1!) machine of mine. Surprisingly, the docile computer is the extremely old PeoplePC-branded box with extremely low resources. (PeoplePC was a prevalent ISP in the mid-1990′s who were, I believe, the first to offer cheap computers as a bonus to new subscribers.)

Unfortunately, this machine runs painfully slowly (especially with a huge application like GNOME) and so even opening windows is an exercise in patience. Doing anything actually productive is simply unrealistic, as it took nearly 2 hours to get the latest updates in up2date from the Red Hat Network, and I actually cancelled the update in the middle of that process. So I had been hoping that Fedora Core 2 would install nicely on the other machines I have.

Of course, that’s when the bad news hit. The bad news is that bugs are in FC2 as well and the same thing happened with it as what happened with FC3 on one of the machines I tried to install it on. I was greeted with that very familiar Install exited abnormally message and was told to reboot my system. I spent a few minutes trying a few of the workarounds I talked about above on this version of Fedora on this machine, but gave up after about half an hour.

Interestingly, on the other machine which didn’t take the install the problem was entirely different. Rather than even getting to the graphical installer, when I pressed Enter at the boot:-prompt I was inexplicably dropped into the text-based installer instead. There was not the slightest indication that the computer was even trying to load the graphical installer. Wierd! This didn’t happen with Fedora Core 3 at all.

Optimistic as I was, I thought it still might work; perhaps it was just the graphical installer that wouldn’t function. But alas, this was not to be. When I was asked where the installation media is to be found and responded with “Local CD-ROM,” the text-based installer reported being unable to locate Fedora Core on the CD-ROM, even though the install disk was right there in the CD drive and I had used that very disc to get FC2 working on the first, oldest machine I tried!

Thus, I have decided that Fedora is a bipolar Linux distribution which needs heavy doses of appropriate patience and medications. I am at a total loss to explain this except to say that I am not left with a great taste for Fedora or Red Hat in my mouth. Ew.

FreeBSD is Less Moody, but also Potentially Confusing

To make matters even more confusing, I had tried to install FreeBSD some days ago on the old machine which now holds the only working Fedora Core installation I have. When I tried to install FreeBSD on it the other day I was unable to proceed due to an error that read something like Unable to locate Hard Disk Drive or similar. I can’t remember the exact wording.

After giving up with Fedora tonight, I tried to install FreeBSD on the machine which forced me into the text-based installer for FC2. It has completed without a hitch, and it only took me a few moments to orient myself within the system (man hier was incredibly helpful) in order to find out how to start a GNOME session and get back to the still-comforting GUI desktop.

Final Fumbling Thoughts

It might be of interest that both machines on which Fedora Core 2 failed to install for one reason or the other are HP Pavillion machines of somewhat different makes. One has a Pentium III processor and the other a Celeron, for instance.

Since FreeBSD has worked on one of these Pavillion towers but not on the old PeoplePC-branded tower (where Fedora 2 did work) I am hoping that at the very least some other BSD will take to the other Pavillion. That way, I can get different operating systems on my various computers and finally get to start playing with the many options out there, like I have been wanting to do for months now.

While I hope this is certainly not the end, I am not at all opposed to the idea of giving up on Linux (or at least Red Hat) for the time being. Perhaps all I really need is to go build myself a new system with new and (hopefully) supported hardware. That really shouldn’t be such a big deal, and I need to upgrade that old PeoplePC box anyway. 200-something mhz is just not acceptable these days. (It wasn’t even in the mid 1990′s, but PeoplePC was cheap and we are not rich.)

I am already far more comfortable with FreeBSD than I was even in the graphical environment of Fedora Core 2′s default GNOME themes. Perhaps it is the shell. I spend a lot of time on Mac OS X within Terminal at a tcsh prompt (though I have playing with bash lately) and, after all, FreeBSD is one of Mac OS X’s parents.

Seems only fitting that I should feel more comfortable with the BSD‘s than with a Linux. In the end, Linux Is Not UniX. ;)