Everything In Between

Without doubt, the most common use of computers today is to create written content of some kind. Blogs are an obvious example, but written content can take a number of forms. Writing manuscripts for publication is another example.

No matter what kind of writing you’re doing, using good tools to make your writing technically better is an incredibly handy thing. Letting the computers do the technical stuff—the stuff they’re good at—let’s you focus on the creative stuff: writing great content. Which is why, if you use a Mac, you’ll be happy to hear that any application’s text field let’s you do a number of really cool things (as long as it’s a Cocoa application, of course).

1. Auto-complete unfinished words

Try this out:

1. Open TextEdit, from your /Applications directory. A new blank document will open.
2. Type Hel and then press the ESC key. A drop-down menu will suddenly appear with an alphabetically sorted auto-complete list of suggestions, sourced from your computer’s current language dictionary. It looks like this:

This feature works with both Pages and, for those of you still using it for some reason (I know you’re out there), TextEdit, too. Also, if you’re a developer and a writer as well (like I am), you’ll be happy to hear that this feature also works with Xcode‘s Code Sense feature, and suggests completions for variable, function, class, and method names in your code.

2. Spellcheck as you type

A Cocoa text input field also has a number of other tricks up its sleeve. For instance, in many applications you can elect to turn on the “Check spelling as you type” feature, which will cause words you misspell (words not in the computer’s dictionary) to appear with a dotted red underline. If you right-click on these words, the contextual menu that appears will offer spelling corrections.

However, sometimes we use words like those in slang or colloquial language that isn’t in a proper dictionary. These words will still appear to be “misspelled” when you type, so in these cases, we can tell Mac OS X to “Learn Spelling” (also from the contextual menu). When you select this option, you append that spelling to your personal dictionary. (This is really just a plain-text file located at ~/Library/Spelling/lang file, where lang is the language code you’re typing in. For Enlgish, this file is ~/Library/Spelling/en.)

3. Look up word definitions and search for text in Google or Spotlight

Last, but certainly not least, another neat thing you can do with text on your Mac is look them up with Dictionary.app. Simply highlight some selectable text on screen, right-click and select “Look up in Dictionary”. This will cause Dictionary.app to open and display the definition of the selected word.

Since Dictionary.app can also look up articles in Wikipedia, this is also a very quick way to go to a Wikipedia article without ever having to open up a Web browser.

Also, from the very same menu, you can open a Web browser. Simply select “Search in Google” to cause your default Web browser to launch a Google search for the highlighted text.

If you’re a Mac user who often uses VPN connections, you’ll notice one very disappointing thing about connecting to your corporate or personal network over such tunneled connections: typically, Bonjour-style addresses (such as “computer-name.local”) don’t work. This is because multicast DNS (or mDNS) doesn’t work over a tunnel. Though there are ways to get it functional, they are pretty complicated and require that you have a lot of esoteric networking knowledge.

However, if the services you typically access via Bonjour use static IP addresses, then there is one age-old networking technique you can use to simulate Bonjour-style naming conventions without actually using Bonjour. This, of course, is the /etc/hosts file.

The /etc/hosts is a simple, static, text-based mapping of computer names to IP addresses. It does exactly what Bonjour does except it doesn’t keep itself up to date when things change. Of course, if you’re using static IPs for the services you want access to, you can pretty safely assume that things aren’t going to be changing frequently anyway. Long-time sysadmins will laugh at this, but I say let them laugh. This is remarkably useful and very easy to implement.

Let’s assume I’m running a personal web server on my home network, and I can access my home network via a VPN. On my home network, my web server’s IP address is, say, 192.168.2.100, and I usually access it as http://server.local/. All I need to do is open a Terminal prompt and run the following commands as an administrative user:

sudo echo "192.168.2.100	server.local" >> /etc/hosts

That’s it. What this does is hard-wire the name server.local so that it always resolves to the IP address 192.168.2.100. Now, anytime anything on my computer tries to access server.local, it’ll always access 192.168.2.100 directly instead of ever needing to make an mDNS query on the network. The net effect is that we can trick our computer into thinking that Bonjour is working, even when it’s not—such as over a VPN connection.

Note that in default cases, hard-wiring an IP address like this completely prevents your computer from ever asking other computers (such as DNS servers) what the current IP address for this name is. That means if the IP address of the remote server changes, you won’t be notified, and things will just not work. So be mindful that you’ve made this change, and revert it as a first step in troubleshooting procedures.

By the way, Windows users can do the very same thing simply by editing their etc/hosts. They can find this file at C:\WINDOWS\system32\drivers\etc\hosts and can edit it with Notepad. They will also need to install Bonjour for Windows to get Bonjour working in the first place, of course.

Using the pbpaste and pbcopy commands, you can manipulate the contents of the Mac OS X clipboard (or more formally known as the pasteboard) right from the command line. As a brief example, just select the text of this first paragraph, copy it to your clipboard (with -c), and then type pbpaste in a Terminal prompt. You should see output similar to the following:

Perseus:~ meitar$ pbpaste
Using the pbpaste and pbcopy commands, you can manipulate the contents of the Mac OS X clipboard (or more formally known as the pasteboard) right from the command line. As a brief example, just select the text of this first paragraph, copy it to your clipboard (with ?-c), and then type pbpaste in a Terminal prompt. You should see output similar to the following:Perseus:~ meitar$ 

Pretty straightforward, right? The only thing to be aware of is that the  symbol showed up in the output as a ? symbol. This is because the Terminal doesn’t support Unicode, but that’s a topic for another time.

Anyway, what’s happening here is nothing more magical than simply reading the clipboard and pasting it into a command’s standard output stream. As a result, you can construct pipelines that read from or add content to the clipboard. Here’s an example in reverse, which takes a command’s standard output and replaces the contents of the clipboard with it:

echo "Hello world! I came from the command line, but now I'm in the clipboard." | pbcopy

This command produces no output, but if we examine the contents of the clipboard (by selecting Edit → Show Clipboard from the Finder’s menu bar) we can see that the text we echoed has indeed been copied there.

Another way we can verify that this worked as expected is to simply pbpaste again:

Perseus:~ meitar$ pbpaste
Hello world! I came from the command line, but now I'm in the clipboard.
Perseus:~ meitar$ 

Anyway, this is cool, but it isn’t very useful yet. For that, we need to add some more stages to our pipeline. Let’s take the simple case of trying to count how many words the selected text contains. We’ll use the clipboard contents shown above to do this:

Perseus:~ meitar$ pbpaste | wc -w
      14

Using the word count (wc) utility, we can count words (-w) very easily. Indeed, our previous example does have exactly 14 words in it. We can also count characters (-c) or lines (-l) of text in the clipboard this way. This is like adding Microsoft Word’s “Word Count” feature to every single piece of text you can copy!

As another example take, for instance, the simple case of copying and pasting a snippet of email from Mail. Instead of pasting it back into a text file verbatim, let’s prepend ‘> ‘ to the beginning of each line. This way, when we paste our email’s snippet, we’ll know where the snippet begins and where it ends. This is a simple three-stage pipeline that uses pbpaste to take our clipboard and put it into the pipeline and then reads back the result from the pipeline back to the clipboard using pbcopy. In the middle, we use sed to insert the desired text at the start of each line:

pbpaste | sed -e 's/^/> /' | pbcopy

Now, when you paste your clipboard, you’ll have a greater-than symbol at the start of each line. Naturally, check out the manual pages for all of these commands for more detailed information. For instance, type man pbpaste for more information about the pbpaste command.

Since Mac OS X 10.2 Jaguar, Mac users have been accustomed to the ease of use of Apple’s very cool Keychain Services technology. The Mac OS X Keychain basically a secure database of all your passwords, sorted into files called (unsurprisingly enough) “keychains.” Each user account on a Mac OS X system has a login.keychain, and the system itself also has a system.keychain.

Whenever you tell an application to “Remember this password in my keychain,” what you’re doing is writing a new encrypted entry into your user account’s ~/Library/Keychains/login.keychain file. Then, the next time the application needs to access a restricted resource, it just asks Mac OS X to get the password for it. Of course, all of this happens automatically, so except for that single checkbox most users probably don’t know that the keychain even exists.

What’s even more awesome than all of this automagic password storing action, though, is the fact that Apple has also provided an easy-to-use application to manipulate the keychain yourself. What good does this do us? Plenty! Observe.

Say you’ve just signed up with a new ISP. They send you a username and a password to log on to their ADSL network with. Of course, they send this password to you on paper—how insecure! Instead, after changing the password to something else first (something other than mypassword, which is the example password I’ll use here), we can use Mac OS X Keychain to securely store the password and retrieve it later.

1. First, launch the Keychain Access application located in the /Applications/Utilities folder of your startup drive.
2. Next, click the “Create a new Keychain item” button (the +) button near the lower left-hand corner of the window. The Add Keychain Item sheet appears.
3. Enter a meaningful name, such as “ADSL ISP Account” in my example, in the Keychain Item Name field.
4. Enter the username or account name associated with this password in the Account Name field.
5. Enter the password into the Password field.
6. Click the Add button.

That’s all there is to it. To later retrieve your password if, say, you ever forget it:

1. Launch the Keychain Access application.
2. Locate and double-click the keychain item that stores the account and password information you want to retrieve.
3. Tick the “Show password” checkbox. You’ll be presented with a dialogue box that asks for your keychain’s master password. Unless you’ve already set it to something else, this is the same password you use to log in to your Mac OS X user account.
4. Enter your keychain password and click “Allow.” If you click “Always Allow” instead, Keychain Access will not prompt you for your login keychain’s password the next time you ask to see this particular password. I never press that button.
5. Your password’s plaintext is now visible.

This effectively obviates the need for third-party applications such as Password Gorilla, PasswordWallet or KeePassX which are great programs, but all suffer from a lack of a good user interface. Furthermore, there’s no reason why we can’t store short arbitrary strings of sensitive information in the keychain temporarily. Sure, it might clutter up your keychain, but you can always search the entries using the standard Mac OS X filter search bar at the top right of the window.

In fact, Apple’s been kind enough to offer an interface to do just that in an even more effective way, called Secure Notes. These are simply plain text strings of arbitrary length that can be stored securely inside your keychain, and that use the same interface to access (requiring your password to view). The only real difference is that instead of a single line, you’re given a fully scrollable text area in which to type your secure note.

Moreover, because keychains can be synced to multiple Macs with .Mac Sync (or a third-party synchronization solution), you can always have access to all your passwords regardless of which physical Mac you’re using. Best of all, since you never have to remember another password ever again, you can quit using the same password for multiple accounts, and you can always use really hard-to-crack passwords.

As I’ve been blogging about, I’ve been playing a lot with Apple’s new WikiServer (or “Teams Server”) at work. We’re still evaluating what we’d like to use it for, but as part of the experiments, I’ve been finding myself having to do some pretty crazy things with the WikiServer. This one is pretty bizarre, and is probably not only very dangerous for the content of your group’s wiki and blog, but also almost certainly not a best-practice.

With that caveat out of the way, here’s how I managed to move all the pages in an Apple WikiServer group wiki and blog from one group to another.

Dear God, please have a backup!

Okay, step 1 is mundane, but seriously, please of please have a backup of your data before you do any of these things. To make a back up of your entire WikiServer’s data store, simply:

sudo tar -cvzf backup-file-name.tgz /Library/Collaboration

If anything goes wrong, you can restore from your backup just as simply:

sudo tar -C /Library/Collaboration -xvzf backup-file-name.tgz

Okay, with that out of the way, next make sure absolutely nobody is using any of the Group services for the group you are going to perform the move from, or to. People can work on other group’s wikis, it’s just the ones you’ll be touching you want people to avoid. You can enforce this with some Apache redirects, which is left as an exercise to the reader.

Step 1: Rename or create a new Group in Workgroup Manager

First, you need to either rename or create a new group in Workgroup Manager. If you make a new group, be sure to enable all the services that the old group used.

If you’re simply renaming a group, then you might not even have to go through this trouble. You merely need to change the group’s “Name” (as opposed to its “Short Name”) and then stop and start the Web Service from Server Admin to see the group’s name change. However, if you also want to change the group’s “Short Name” (i.e., the group’s POSIX group account symbolic name), then you will need to perform these steps.

Once you have the new group ready to go in Workgroup Manager and you have stopped and started your Web Service in Server Admin, continue to the next step.

Step 2: rsync all your files from the old group to the new group

This is simple. Just run:

sudo rsync -avzE --progress /Library/Collaboration/Groups/old-group/ /Library/Collaboration/Groups/new-group/

Note that the trailing slashes on the directory names in this command are quite important, as they tell rsync to take the contents of the first directory and place those items as the contents of the second directory. Without the trailing slashes, rsync will make extraneous directories for you, which Apple WikiServer won’t understand. See man rsync for more information.

Step 3: Update the plists for all your WikiServer pages in the new group

Now, WikiServer has all the content of your old group but all of the internal references are wrong, since they still point to the old group. What you need to do is rewrite all those references so that they point to the new group address. The group references are stored in property list files. See my older blog entries for details about the filesystem structure of Apple’s WikiServer data storage layout.

To do this is a relatively simple procedure. As root, do the following, and do it carefully because as root you can easily mess up (and there is no undo button on the command line!):

sudo su -
cd /Library/Collaboration/Groups/new-group
grep -ri "old-group" * | grep '^[^B]' | cut -d ':' -f 1 | grep 'plist$' | sort | uniq > /tmp/list_of_plist_files_to_edit
for i in `cat /tmp/list_of_plist_files_to_edit`; do sed -e 's/<string>groups\/old-group/<string>groups\/new-group/' $i > $i.new; mv "$i.new" "$i"; chown teamsserver:teamsserver "$i"; chmod o-rwx "$i" done;

In English, this means:

1. Become root.
2. Change to the /Library/Collaboration/Groups/new-group directory
3. Find all plist files that have the the string old-group in them, sort them, and write this list of files to the /tmp/list_of_plist_files_to_edit file.
4. For each of the files listed in the /tmp/list_of_plist_files_to_edit, find the text string <string>groups/old-group and replace that text with <string>groups/new-group, and save this change as the name of the file with .new appended. Finally, replace the original file with the file we modified, and give them the appropriate permissions.

Step 4: Create an Apache redirect to make sure no HTML links are broken

Okay, at this point your new group is up and running and it should be working. However, if you had any links at all in any of your group’s pages, they are now all broken because they still point to the old group. Rather than going through the HTML itself and cleaning this up right now (because that’s very error-prone indeed, even with automated tools), it’s much easier to just tell Apache to redirect all requests for the old group to the new group.

To do this, edit the Apache configuration file of whatever Virtual Host you have been serving the WikiServer from. Most of the time, this will be at /etc/httpd/sites/0000_any_80_.conf.

The end of that file probably looks something like this::

#       Include /etc/httpd/httpd_users.conf
#       Include /etc/httpd/httpd_directory.conf
        Include /etc/httpd/httpd_groups.conf
        Include /etc/httpd/httpd_teams_required.conf
        LogLevel warn
        ServerAlias *
</VirtualHost>

Right above these Include directives, simply add the following:

        <Location /groups>
            <IfModule mod_alias.c>
                Redirect 301 /groups/old-group http://your.server.address/groups/new-group
            </IfModule>
        </Location>
#       Include /etc/httpd/httpd_users.conf
#       Include /etc/httpd/httpd_directory.conf
        Include /etc/httpd/httpd_groups.conf
        Include /etc/httpd/httpd_teams_required.conf
        LogLevel warn
        ServerAlias *
</VirtualHost>

There you have it. New Teams Server group, old group’s data.

Caveats

Note that this does not update any of the SQLite databases used to store things like revision history and so forth. These things are, for the most part, not really necessary to update but it would be ideal if the old plist revisions could be changed in there, too. That’s not so much more extra work, really, but I’ve found it typically unnecessary except in fringe cases, so I leave that as an exercise to the reader.

If you do this to your WikiServer and do not update the SQLite databases as well, just be mindful of that fact so that you’re not surprised if something goes wonky down the line.

This should be straightforward, but it’s not. To import (not track, but just import) code from a remote CVS repository to a local git repository, you need to do the following:

1. Be certain you have the git-core package installed on your system and that this package includes the git-cvsimport command. You can run git help -a | grep cvsimport to verify this.
2. Be certain you have the cvsps command-line tool installed. This does not come with the git suite of tools, so you’ll need to get it separately. If you’re a lazy Mac OS X user, like me, you can use MacPorts: sudo port install cvsps. Otherwise, get it from the source.
3. Prepare your CVS login information for the remote server before you run git cvsimport. You need to do this so that the git tool will be able to log you in to the CVS server automatically. The command for this looks like:

   CVSROOT=:cvs-login-method:cvs-user-name@cvs.server.name:/path/to/CVS/root cvs login

   For example, if you’re pulling code from the anonymous CVS server that runs on Drupal.org, you might use this: CVSROOT=:pserver:anonymous@cvs.drupal.org:/cvs/drupal-contrib cvs login. This command will prompt you for the password for the user you specified at the server you specified (for anonymous access, the password is almost always anonymous) and will hash this in the ~/.cvspass file for future use by CVS

4. Finally, run the git cvsimport tool, and specify the proper options. Using the Drupal example above, your command might look like this:

   git cvsimport -v -d :pserver:anonymous@cvs.drupal.org:/cvs/drupal-contrib contributions/modules/module-name

   This would login to cvs.drupal.org using the CVS‘s pserver login method, provide the username anonymous and the password you specified in the previous step that is hashed in ~/.cvspass, set the CVS document root to /cvs/drupal-contrib, and pull the code located at contributions/modules/module-name into the current working directory as a git repository.

This works pretty nicely, and creates a git repository just as though you’d created it with git init in the current working directory.

If you get an error that looks like this:

AuthReply: cvs [pserver aborted]: descramble: unknown scrambling method

then you’ve most likely specified the CVS document root incorrectly. Most notably, git cvsimport does not understand a CVS document root wherein the password is specified in the document root URL itself. So, for example, git cvsimport -d :pserver:password:username@cvs.server.name:/path/to/CVS/root code/to/checkout will not work. Omitting the password and the separating colon from the URL should fix it.

Last time, we checked out the Apple WikiServer from the user’s side of things. We learned about the code it generates, how it handles page name changes, and what the key filesystem locations the Apple WikiServer looks at are. This time, let’s delve a little deeper into the WikiServer’s internals by (safely) messing around with a running instance and seeing how we can effect changes that aren’t available in the GUI.

I’m going in!

Apple’s WikiServer keeps each group’s data segregated. There’s currently no way in the GUI to search across multiple wikis from a single wiki, or move one wiki page in one wiki to another wiki. Luckily, because the Apple WikiServer is a rather lightweight application, it’s not that hard to move things around. Doing so doesn’t seem to cause any problems with the application, either—a testament to its apparent simplicity.

Most simply, say you have created a number of different wikis for different groups, and in so doing you’ve created segregated spaces for different parts of your organization. Now say you’ve got a wiki page in one wiki that you’d rather place in a different one, or that you’d like to copy. Sure, you can edit the first wiki’s page, copy the WYSIWYG editor’s text, and paste it into a new page on the second wiki. But where’s the fun in that?

Instead, let’s move some files around to get a sense for how the Apple WikiServer stores its data.

First, recall that the Apple WikiServer stores its content inside the /Library/Collaboration directory on the computer hosting the wiki web site. Inside of this directory you’ll find four items:

$ cd /Library/Collaboration/
$ ls -la
total 8
drwxrwxr-x   6 _teamsserver  _teamsserver   204 Mar  2 13:50 .
drwxrwxr-t@ 60 root          admin         2040 Mar  3 19:16 ..
drwxr-xr-x   2 _teamsserver  _teamsserver    68 Mar  2 13:50 ArchivedGroups
drwxr-xr-x   6 _teamsserver  _teamsserver   204 Apr  4 18:48 Groups
drwxr-xr-x   2 _teamsserver  _teamsserver    68 Mar  2 13:50 Users
-rw-r--r--   1 _teamsserver  _teamsserver   249 Mar  2 13:50 dataVersion.plist

First, note that all of these files are owned solely by the _teamserver user, and only that user can access these files in any way. This keeps people like you and me from accidentally tampering with the Apple WikiServer data store. However, for the purposes of this exercise, we’re going to tamper anyway. To do so, you’ll need root permissions, which comes with a big caveat: be careful whenever you are root—one typo could erase your hard drive (and there is no undo)!

Become root by saying:

sudo su -

and supplying your password. Once you are root, you can explore further into the depths of the Apple WikiServer.

Inside the Apple WikiServer data store for wiki pages

The Apple WikiServer, which runs under the username _teamserver, is really just a bunch of organized flat files, supplemented with a number of SQLite databases (also flat files, really), and glued together with a bunch of Python scripts running under the Twisted networking framework. We’re not going to cover Python or Twisted in this post, so for further information check out their respective homepages. There’s also a great introduction to Twisted at ONLAMP.com.

To make the wiki go, the WikiServer runs these scripts to read and write data to these flat files. Let’s dig into this data store. Since the wikis are each associated with an existing group, all of the wiki files are inside the Groups subdirectory. In this directory, you’ll find one directory for each group that has a wiki. Each group directory, in turn, contains 6 other directories, 2 regular files, and 1 SQLite database:

# pwd
/Library/Collaboration/Groups/chiefs
# ls -p
discussion/	index.db	metadata.plist	resources/	wiki/
extrainfo	mailinglist/	public/		weblog/

The two regular files are metadata.plist and extrainfo. The metadata.plist file maintains the preferences of the individual wiki web site that you set from the “Group Settings” page on the wiki web site itself. That is, these settings are all editable via the web directly at http://your-server.local/groups/your-group-name/settings/. That page is also accessible to an admin user from the wiki home page under the “Admin functions” header in the sidebar of the web page (the key names in the plist file are pretty self-explanatory and map easily to the GUI options).

At the moment, the extrainfo file is a bit of mystery. It stores single lines of plain ASCII text in the form:

groups/your-group-name/web-service/page-id

where your-group-name is the name of the associated group for the file, web-service is one of wiki, weblog, and I assume can also be mailinglist or discussion, although I’ve only seen the first two on my server in practice.

What we’re most interested in for the purposes of wiki pages is, unsurprisingly, the wiki directory. This directory stores as many subdirectories as there are pages on your wiki. Every time you create a new page, a new directory is added here. If you delete that page, however, this directory is not deleted outright.

Each directory is named with the unique page identifier that the Apple WikiServer automatically generates followed by a .page extension. The exception is the welcome.page directory, special in that it contains the content of the wiki’s main (front) page. The Mac OS X Server Web Technologies Administration for Version 10.5 Leopard manual explains the contents of this directory as follows:

• /Library/Collaboration/Groups/groupname/wiki/pagename.page/ contains the component files of a wiki page.
• /Library/Collaboration/Groups/groupname/wiki/pagename.page/page.html contains the main text of the wiki (html content).
• /Library/Collaboration/Groups/groupname/wiki/pagename.page/page.plist contains the metadata for the wiki page.
• /Library/Collaboration/Groups/groupname/wiki/pagename.page/revisions.db contains the version history database for that wiki page.
• /Library/Collaboration/Groups/groupname/pagename.page/images/ contains the images for that wiki page.
• /Library/Collaboration/Groups/groupname/pagename.page/attachments/ contains all attachments for that wiki page.

Here are some additional details:

• The “page.html” file contains all of the (X)HTML that appears inside the WYSIWYG editor field when you click the “Switch to HTML View” button (the arrow brackets). Nothing more, and nothing less.
• The “page.plist” file maintains the metadata that associates a single wiki page to its group as far as the Apple WikiServer is concerned. If you take a closer look at it, you’ll find property list keys such as commentUID and uid whose values are both strings—URLs, in fact—that the Apple WikiServer uses to, for example, call the appropriate wiki theme to render around the page content. This file also stores self-explanatory data such as author, createdDate, page kind, a last modifiedDate and lastModifiedAuthor and, of course, the page title. The most interesting key in this file is the versioned key, which is a boolean that can turn versioning on or off. By default, wiki pages have this key set to true and (for example) weblog pages have this set to false. However, you can manually turn off versioning for a specific wiki page by changing this key yourself. It won’t delete past revisions, it will just stop new revisions from being saved. This is especially cool, because it means you can version-control weblog pages simply by editing the page’s associated plist file.
• The revisions.db is the SQLite database in which all revisions are stored. The database contains a single table, revisions, that looks like this:

  CREATE TABLE revisions(revision INTEGER PRIMARY KEY AUTOINCREMENT, editType, comment, lastModifiedBy, content, time DEFAULT CURRENT_TIMESTAMP);

  The interesting bit is the content field, which itself is just a string—but a string stored as an Apple property list, in full. Some keys in the property list inside the SQLite database are the same as those inside the page.plist file for the page, but with some differences. For instance, there is no versioned key in the plist in the database (for obvious reasons), and there is a tags key (which is missing from the filesystem equivalent). Most importantly, there is a content key, which stores the content of the page.html file.

• The revisions.db file is created automatically only if the versioned key in the page’s page.plist file is set to true.

Finally: move a wiki page from one wiki to another

Now that we understand some of the internals of the Apple WikiServer data store, let’s mess with it just a little by moving that one test wiki page from one wiki into another. Since almost everything is flat files, it’s remarkably easy to do:

# mv /Library/Collaboration/Groups/your-group-name/wiki/page-id.page /Library/Collaboration/Groups/new-group-name/

Be certain that there isn’t already an existing page in the new-group-name wiki with the same page-id as the one you’re moving. So far, Apple WikiServer appears to generate unique page IDs across the entire system (as opposed to just per-group), but I haven’t verified this yet.

With the wiki page files physically moved into the new wiki, you can now access the page at the new wiki URL. Trying to access it in the old URL will result in an HTTP 404 “Not Found” error. However, the page at the new wiki URL has the same theme as the other, old wiki. To correct this, you need to edit the page.plist file in two places: the uid string value and the commentUID string value relative URLs need their your-group-name changed to the new-group-name, whatever it is.

After you save this this change, simply reload the wiki page in your browser. Voila. Page moved.

Lately, there have been a string of networking problems with Mac OS X 10.5 Leopard reported by sites such as MacFixIt. One of most common symptoms is the loss of Bonjour’s “.local” addresses. So, for instance, if you have a machine named “Perseus” then you could address that machine by the hostname “Perseus.local” instead of its IP address.

However, if you find that the .local host name no longer works but the IP address still does, the problem may be in a corrupt or outdated local DNS cache. Luckily, the solution is incredibly simple. Just run:

dscacheutil -flushcache

at a Terminal prompt, and try again. If you’re still running Mac OS X 10.4 Tiger and you experience this problem, the solution is just as simple. Instead of the above line, simply run:

lookupd -flushcache

at a Terminal prompt.

I absolutely love wikis, so when Apple introduced Mac OS X Server 10.5 “Leopard,” one of the new features I was really excited about was “WikiServer” (what the Apple marketing department calls “Teams”). I’m calling this specifically the Apple WikiServer in order to avoid confusion with the pre-existing wiki plus web server package called WikiServer.

Apple WikiServer: Mac OS X Server’s built-in Intranet builder

At work, I’m finally getting the opportunity to try the Apple WikiServer out. Its strongest asset, by far, is the integration it has with Apple’s Mac OS X permission scheme. Apple WikiServer makes heavy use of the OS‘s built-in user accounts to define users and groups, and the permissions those users and groups have to edit, view, and comment on pages in the wiki you create with it. And, because Leopard Server has full support for Access Control Lists, those permissions schemes can be as complex as you like.

This is very important because many large (and small) organizations have sensitive material that they’d like to keep private, or restricted to certain groups. Historically, wikis are a free-for-all. Anyone and everyone who has access to any part of the wiki can change any other part of the wiki. Recent wiki implementations such as later version of MediaWiki and some other wiki software have implemented permissions systems to allow administrative users to control access rights, but these are often complicated or require code-level configurations.

With Apple’s WikiServer, all of these permissions can be managed via the Workgroup Manager application, and because you can take advantage of the built-in ACL support, you can model your organizations permissions scheme directly in the Server OS permissions structure, giving you a much easier way to control information access. Take note, however, that like almost all other things that have to do with your Apache configurations, your Server’s Web Service will likely need to be stopped and started again for any changes you make to a wiki’s permissions take effect.

The Workgroup Manager application is also where you go to create new wikis for groups. To enable a wiki, you need to already have created a group and assigned users to that group. For instance, I created a Developer Wiki where all of the in-house developers can share tech tips, so I created a group called “Developers” and assigned individual developers, as well as the company executives (by way of the “Executives” group) to that group. The group-within-a-group technique is key, because if the company executives change, the members of the Developers group does not need to change, too. In all of Apple’s publications, Apple refers to the wikis hosted by WikiServer as an “intranet website.”

It’s clear that Apple intended this product for use within small companies, and not necessarily out on the open Internet. What follows are just a few notes I’ve compiled about how the Apple WikiServer works.

Front-end Code Generation from the Apple WikiServer WYSIWYG Editor

The Apple wikis are very nice to use. Their functionality is relatively straightforward to find and activate. However, the HTML code that the Apple wikis generate can be a little confusing. By default, new page text is entered into a semantically meaningless <div> element. This can be changed by highlighting text and then selecting “Paragraph” from the formatting toolbar. Subsequent paragraphs that are typed seem to then use <p> elements. However, some paragraphs revert back to <div>s when I used it, and I’m still not sure why or when this occurred.

On the plus side, so far, all the browsers I’ve used with the Apple WikiServer function the same way. This include Firefox 2, Safari 3.1, and Internet Explorer 6 and 7.

Typing actual code and having it marked up as such can’t be done in the GUI formatting toolbar to select a <code> element. The “Monospace” item in the text formatting toolbar creates <pre> elements and <pre> elements only. However, Apple does provide a “Switch to HTML view” button (the arrow brackets button) and one can enter standard HTML, including <code>…</code> elements in that view, and then switch back. This behaves perfectly on all browsers except Internet Explorer, in which your text area field shows no line breaks whatsoever.

Apple’s WYSIWYG editor handles escaping special characters when those special characters have HTML entity reference equivalents, such as double quotes (“), arrow brackets (< and >), and ampersands (&). It does not seem to handle Unicode characters, such as the ellipses in the prior paragraph. However, such Unicode characters need not be escaped as long as the document’s character set is UTF-8 (or UTF-16), which the Apple WikiServer specifies and supports out of the box.

Pressing the Return key twice causes the Apple wiki to generate an empty <div> or <p> element with an explicit break (<br />) inside of it. One can deduce that this is a design choice in order to help transition users who are used to plain <textarea /> inputs to Apple’s WYSIWYG editor. It’s also the only way to space paragraphs properly if the user hasn’t selected the “Paragraph” option in the text formatting toolbar. Otherwise, simply hitting the Return key once is enough to space paragraphs apart properly (i.e., the functionality is equivalent to the way Microsoft Word or Pages handles paragraph breaks).

Interestingly, the “Enter URL…” functionality form the toolbar is smarter than one might first assume. For instance, it recognizes email address and prepends a mailto: scheme to the link if it finds one. This is contrary to what the Apple-provided manual states, which tells you to enter the “mailto:” portion as part of the URL. In fact, you should omit this, else your final mailto link will actually read “mailto:mailto:your.email@address.com”.

This means linking to “mailto: links” is as simple as typing an email address. Similarly, the WYSIWYG doesn’t complain if your fully-qualified URL doesn’t include a scheme, so you can enter //apple.com/ and the subsequent link is generated as <a href="//apple.com/">Link text</a>. This is one step above and beyond even WordPress’s new WYSIWYG editor, which forcefully prepends an http: scheme to URLs without one.

For the most part, copy-and-paste works as expected, except in cases where the WYSIWYG editor does not understand the current formatting, such as a specific font and (and this is a biggie) for links. At first, the editor will appear to show that the formatting (including links) is saved, but when you actually save the page, only the formatting that the WYSIWYG editor understands is actually saved. Worse, all your links are turned into underlined—but unlinked—text. In short, this means that if you are copying and pasting page content that contains links, you need to do so in the HTML view of the page editor.

Inexplicably, the editor generates <i> and <b> tags for italics and bold, instead of the preferred <em> and <strong> elements. I’m not sure I understand why this is the case. There does exist a an “emphasis” option in the toolbar, as does an “important” option, but these generate strange spans instead. The “Important” item wraps the selected text inside a <span class=”Apple-style-span custom_forecolor_important”>…</span> and the “Emphasis” item wraps the selected text inside a <span class=”Apple-style-span custom_forecolor_emphasis”>…</span> element.

There’s also one other item, “Highlight,” which wraps the selected text inside of a <span class=”Apple-style-span custom_backcolor_highlight”>…</span> element.

The only explanation that makes sense to me, after much speculation, is that perhaps Apple does not want to encode semantic information such as what <em> and <strong> would imply from users who use a WYSIWYG editor. This shows either a blatant distrust of users or incredible foresight. I’m not sure which.

Page names and URLs

Currently, the search functionality built into Apple’s WikiServer only searches on the text of a page’s title.

Apple’s WikiServer generates unique page names for each new wiki page. These names consist of two parts, but only one seems to make any difference. For example, if you create a new Wiki page called “Hello”, you might get an address in your web browser’s location bar that looks like this:

http://your-server.local/groups/your-group-name/wiki/1d06a/Hello.html

Most of this is standard URL stuff (the protocol, the server address), and most of the rest is self-explanatory (the group name, the wiki section). The important bits in this URL are the last two:

• 1d06a
• Hello.html

Obviously, “Hello.html” came from the fact that you named your new page “Hello”. WikiServer appended “.html” on its own. The other bit, the short string of random characters, is a unique identifier used across all of this group’s web services (the wiki, blog, calendar, and mail list, if these other services are enabled) to uniquely identify this page. What’s interesting about this is that only the unique string of characters seems to matter in regards to accessing the page. That is, if you next ask for:

http://your-server.local/groups/your-group-name/wiki/1d06a/Some-Random-Page.html

you’ll still get the same “Hello.html” page from before, even though you’re seemingly asking for “Some-Random-Page.html”. In fact, it doesn’t seem to matter what you replace “Some-Random-Page” with. So long as there is some text in that part of the URL, that the URL ends with “.html” and that the unique identifier remains untouched, you’ll always end up retrieving the “Hello.html” page.

This means that if you change this page’s name later to, for instance, “Hello world”, old links that point to “…/1d06a/Hello.html” will continue to work, even while new links will start to point at “…/1d06a/Hello_world.html”. From a usability perspective, this is simple and effective; it ensures that users have a reminder of what the page is about by looking at the last part of the URL. However, once page names change, it becomes a bit non-optimal, because the same page can be referred to by multiple names—a “no-no” in the SEO world and a practice discouraged by most semantic-web types.

I would imagine that Apple made this design decision because the company envisioned their WikiServer to be used, again, primarily in intranet and SOHO environments, and as a result are not too concerned with search engine optimization. As an aside, this unique string is also how Apple’s WikiServer identifies the stored content on the filesystem. Read on for more details.

Hacking the Apple WikiServer

There isn’t a lot of information out on the Web right now about how to work with the WikiServer, especially for developers. Therefore, some digging is needed. After a bit of research, I discovered the following key directories that the WikiServer uses. They are as follows:

• /usr/share/collaboration - This has a few developer tool support files as well as the majority of the client-side code for the Wiki (javascripts, etc).
• /usr/share/wikid - This directory holds the Python sources and compiled bytecode for all the “Teams” components (including wiki, blog, calendar, etc.). It seems to run on Twisted and a number of other familiar-sounding components.
• /Library/Application Support/Apple/WikiServer - This is where most the data is stored, inside of plist files and a few others. The Themes subdirectory here is where Apple recommends that look-and-feel changes be made.
• /Library/Collaboration – This is the default data storage location for all the “Teams” components. The actual content of the wikis and blogs will be kept somewhere in this directory, which means that this is the directory you want to backup to backup the content of your wikis. This location is the only user-configurable one of the bunch. To change it, change the “Data Store” value in Server Admin. (A more detailed listing of this directory hierarchy is available on page 62 of the Mac OS X Server Web Technologies Administration For Version 10.5 Leopard manual.)

If you take a peek at the /Library/Collaboration directory, and follow that into the Groups/your-group-name/wiki directory, you’ll find a list of all the pages in your wiki stored as .page bundles, identified with the unique character string WikiServer generated when it first created the page.

It should be noted that anything in the /usr/share directory will likely be overridden whenever Apple releases an update that modifies the WikiServer. As a result, any and all changes you make to WikiServer’s templates or themes should be done by creating new files in the /Library/Application Support/Apple/WikiServer/Themes directory.

It’s interesting to note that the WikiServer seems to use Python for its back-end processing. This may open up some interesting integration possibilities for Python programmers in the future.

More help eslewhere

Even though Apple WikiServer is relatively new, there’s a load of helpful information about it on the web. Most of the good stuff is on Apple’s own Discussions boards, but more and more info is beginning to show up on blog posts. A Google search should give you what you need. For the really lazy, however, here are a few helpful items:

• Server Admin 10.5 Help: Customizing Wiki Themes and Layouts
• Apple Discussions: How to customize Apple WikiServer templates and themes
• Mac OS X Server Web Technologies Administration For Version 10.5 Leopard

This was just a brief introduction to WikiServer from some notes I’ve been collecting in my experimentations, but I hope it’s helpful to someone somewhere. Cheers. Or, continue to Part 2.

In some situations, like the odd one I now find myself in, the only way to get Internet connectivity is to use a solution that requires a fair bit of maneuvering. In my situation, I have temporarily obtained a Vodafone 3G mobile card. Unfortunately, the Vodafone Mobile Connect software for Mac OS X as of this writing is obscenely poor. Of course, Vodafone’s software for Windows works without a hitch.

The only way I could get my Vodafone 3G card to work was to fire up a Windows XP guest inside of my MacBook Pro, using VMware Fusion. Connecting to the Internet with the 3G card using the Windows guest was smooth sailing, but that only provided the Internet connection to the Windows virtual machine. I wanted my Mac to be directly connected.

The solution is obvious, but a few gotchas really bit me hard. To get the Windows guest to share its Internet connection from the 3G card to my Mac, I would need to bridge VMware’s virtual ethernet adapter from the Windows guest to the Mac OS X host. Once bridged, both the Windows guest and the Mac OS X host would logically be on the same ethernet network segment. At this point, I can enable Windows XP’s built-in Internet Connection Sharing (stupidly dubbed “ICS” because everything needs a TLA) on the 3G connection so that Windows NATs it through to the bridged virtual ethernet card. Finally, I can connect to Vodafone’s 3G network, and all should be well.

Here’s the gotchas.

First, in order for VMware to actually initiate the network bridge when it starts up, it must detect that a physical link is active on your Mac. In other words, Mac OS X’s Network System Preferences pane must show you a yellow dot next to at least one physical networking device (probably either your “Built-in Ethernet” or your “AirPort” ports). VMware Fusion will give you no errors or warnings that a bridge is unavailable until you try to connect your virtual machine’s network while set to bridge, in which case VMware Fusion will complain with an error that reads: “The device on /dev/vmnet0 is not running.”

Obviously, if you have no other devices to connect to, you need to fake one. The easiest way to do this is to set up a Computer-to-Computer network using AirPort. Just go to your AirPort menu bar item and select “Create Network…” and create the network (preferably encrypted). If you check System Preferences now, you should see a that AirPort has a yellow dot next to it and reads as having a “Self-Assigned IP Address.” Now that you have a physical link on your AirPort card, you should be able to start the VMware Fusion virtual machine with bridged networking mode without incident.

However, if you do encounter the above error anyway, you need to restart the VMware network bridge. You can do this either by shutting down VMware completely (turn off your guest operating systems, and quit the VMware Fusion application), or you can run the following commands as an administrator in Terminal, which will stop any bridge currently running (or do nothing if no bridge is running) and then restart it, providing the output as shown:

sudo killall vmnet-bridge
sudo "/Library/Application Support/VMware Fusion/vmnet-bridge" -D vmnet0 ''
Entering event loop...
Examining network configuration...
Turning on bridge with host network interface en1...

Obviously, you may be asked for your password as you perform this procedure. Note that the trailing two apostrophes are single quotes with no space. This is (almost) how the VMware Fusion boot.sh script starts and stops the network bridge. Specifically, you’re telling the vmnet-bridge application to run in Debug mode and to bridge vmnet0 to whatever is the current primary networking interface. In the example output shown above, this is en1, or my AirPort card connected to the computer-to-computer network I created in the previous step.

Hopefully you won’t have to mess with the vmnet-bridge application, as this should happen on its own when you start up VMware Fusion if you have any physical link on a network device. Nevertheless, I’ve found this is sometimes unreliable, so just in case it doesn’t now you know how to bring up the bridge on your own. (Tip: once it’s up, you can CTRL-Z to pause it, re-start it with fg %1 and then quit Terminal if you like. The bridge will still be up.)

Now that the AirPort card has a physical link, and the VMware network bridge is running, the next step is to configure your virtual machine to use bridged networking. Just go to Virtual Machine → Network → Bridged as normal. Make sure Connected is also selected. Now start up your Windows guest.

Once Windows boots, go to the Network Connections window by selecting Start → Connections → Show all connections. At this point, your “Local Area Connection” in Windows probably has a warning sign on it and reads as having “Little or no connectivity.” It probably has a self-assigned IP address just like your AirPort card. That’s fine—as long as it’s not “unplugged,” we’re in good shape.

Next, select whatever other connection you want to share the Internet from (in my case, the 3G modem, but it could also just be any other connection in the window), right-click it and select Properties. Go to the Advanced tab and make sure “Allow other network users to connect through this computer’s Internet connection” is checked. The other boxes won’t matter.

What this does is turns on Windows’ own NAT service that configures the one connection (the one your sharing) as the WAN side of (yet another) virtual networking device and the Local Area Connection (the one we’ve bridged to our AirPort or Built-in Ethernet card on our Mac) as the LAN side. Hit OK as many times as is necessary to close the network connection properties windows and wait a few moments. Sometimes this can take up to 30 seconds or so, but eventually you’ll see Windows announce that “Local Area Connection is now connected.” If you inspect it, you’ll see that the IP address configuration has been automatically assigned as a “Manual Configuration” with the address of 192.168.0.1, a subnet mask of 255.255.255.0, and no default gateway.

As a last step, now we can actually connect to the Internet using whatever service we have. In my case, this is when I hit the “connect” button on my Vodafone Mobile Connect software. Once the connection is established and the Windows XP virtual machine can see Internet, it takes up to another minute or two (or three) for the Mac’s connection to get an IP address from the Windows guest, but it invariably works.

If the Windows side of things is giving you any trouble, the most reliable solution I’ve found is to simply disable, then re-enable whatever connection isn’t behaving as desired. If after all of this your Mac still doesn’t get an IP address from the Windows XP guest, disconnect and then re-connect the virtual machine’s ethernet card (by toggling the “Connected” menu item in the Virtual Machine → Network menu). Also, of course, be doubly sure that your AirPort is set to “Use DHCP.”

Phew! So simple…and yet so much harder than it had to be. I found the following two PDF documents very helpful in understanding all of this. You might too:

1. VMware Fusion Network Settings — a super-brief, but excellent introduction to VMware’s network setting internals. It’s also a PDF download attached to the linked forum thread.
2. Share Windows XP Guest Internet Connection with OS X Host HOWTO — This basically describes the same thing this post does, but it does so using absolute step-by-step instructions. It’s also a PDF download attached to the linked forum thread.