Everything In Between

The brutally honest, first-person account of Meitar Moscovitz's life.

Archive for April, 2008

Using Calendars from the Command Line

2 comments

If you’re anything like me, you always have a terminal window open. One of the reasons I do this, of course, is because it’s fast. If I want to know anything at all about my computer, all I need do is type the question. The answer, because it’s always text-based, comes back immediately. I don’t have to wait for a window to open or for a pane to scroll. Everything comes at me from a single visual direction, the bottom of my terminal window.

However, there are some occasions when a text-based response to a complicated question isn’t very helpful because it requires so much extra work to understand. For me, the most common example of this sort of issue has always been in looking at time-based information, and more specifically, calendars. Whenever I’m on my machine, I almost always need to look at a calendar.

In the past, I used to go all the way over to iCal. Sure, I can do this using keyboard shortcuts only, but sometimes all I want is a quick answer to “what date is this upcoming Friday?” In situations like that, I’ve lately begun using the cal command, and my oh my, what a timesaver.

cal is kind of like man for dates. Of course, you can get more info by saying man cal to your prompt. The cal program, installed by default on almost all UNIX-based systems (including Mac OS X), has a ton of useful options. However, most of the time, I don’t need more than a few.

For instance, let’s say I just want a calendar of the current month. I can get get a compact, simple month view instead of going to iCal by saying just cal at the command line:

Perseus:~ maymay$ cal
     April 2008
Su Mo Tu We Th Fr Sa
       1  2  3  4  5
 6  7  8  9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30

Other options let me ask other questions of cal. Easy, simple, fast. I like it.

Written by Meitar

April 18th, 2008 at 8:47 pm

How to import CVS code repositories into Git using `git cvsimport`

9 comments

This should be straightforward, but it’s not. To import (not track, but just import) code from a remote CVS repository to a local git repository, you need to do the following:

  1. Be certain you have the git-core package installed on your system and that this package includes the git-cvsimport command. You can run git help -a | grep cvsimport to verify this.
  2. Be certain you have the cvsps command-line tool installed. This does not come with the git suite of tools, so you’ll need to get it separately. If you’re a lazy Mac OS X user, like me, you can use MacPorts: sudo port install cvsps. Otherwise, get it from the source.
  3. Prepare your CVS login information for the remote server before you run git cvsimport. You need to do this so that the git tool will be able to log you in to the CVS server automatically. The command for this looks like:
    CVSROOT=:cvs-login-method:cvs-user-name@cvs.server.name:/path/to/CVS/root cvs login

    For example, if you’re pulling code from the anonymous CVS server that runs on Drupal.org, you might use this: CVSROOT=:pserver:anonymous@cvs.drupal.org:/cvs/drupal-contrib cvs login. This command will prompt you for the password for the user you specified at the server you specified (for anonymous access, the password is almost always anonymous) and will hash this in the ~/.cvspass file for future use by CVS

  4. Finally, run the git cvsimport tool, and specify the proper options. Using the Drupal example above, your command might look like this:
    git cvsimport -v -d :pserver:anonymous@cvs.drupal.org:/cvs/drupal-contrib contributions/modules/module-name

    This would login to cvs.drupal.org using the CVS‘s pserver login method, provide the username anonymous and the password you specified in the previous step that is hashed in ~/.cvspass, set the CVS document root to /cvs/drupal-contrib, and pull the code located at contributions/modules/module-name into the current working directory as a git repository.

This works pretty nicely, and creates a git repository just as though you’d created it with git init in the current working directory.

If you get an error that looks like this:

AuthReply: cvs [pserver aborted]: descramble: unknown scrambling method

then you’ve most likely specified the CVS document root incorrectly. Most notably, git cvsimport does not understand a CVS document root wherein the password is specified in the document root URL itself. So, for example, git cvsimport -d :pserver:password:username@cvs.server.name:/path/to/CVS/root code/to/checkout will not work. Omitting the password and the separating colon from the URL should fix it.

Written by Meitar

April 15th, 2008 at 4:06 am

Posted in HOWTO,Mac OS X,Programming,Unix/Linux

Tagged with

Reason Number…uhh…Why I Love VoIP

one comment

So, even though I’m still fighting with the Customer Service department of iiNet to realign my billing cycle with the day (over two weeks late) of when my Internet service actually started, I have to say that the service itself is superb. It may just be because I’m in prime DSL location, only a building away from the DSLAM telephone exchange for my area. Still…I’m getting better Internet speeds than I ever got from Time Warner Cable and Road Runner in New York City. Of course, I am paying a bit more for it, but the quality makes up for the cost for what I’m used to.

Even better, iiNet’s naked DSL service comes with a bundled VoIP plan. This is just as awesome as the Internet service. International calls, I discovered, are only 6 cents a minute to the United States. This means I have options like Skype and really cheap VoIP calls. Not bad for a country in the “technological stone age.”

Written by Meitar

April 13th, 2008 at 6:09 am

Posted in Personal

A web developer’s introduction to the Apple WikiServer (part 2)

3 comments

Last time, we checked out the Apple WikiServer from the user’s side of things. We learned about the code it generates, how it handles page name changes, and what the key filesystem locations the Apple WikiServer looks at are. This time, let’s delve a little deeper into the WikiServer’s internals by (safely) messing around with a running instance and seeing how we can effect changes that aren’t available in the GUI.

I’m going in!

Apple’s WikiServer keeps each group’s data segregated. There’s currently no way in the GUI to search across multiple wikis from a single wiki, or move one wiki page in one wiki to another wiki. Luckily, because the Apple WikiServer is a rather lightweight application, it’s not that hard to move things around. Doing so doesn’t seem to cause any problems with the application, either—a testament to its apparent simplicity.

Most simply, say you have created a number of different wikis for different groups, and in so doing you’ve created segregated spaces for different parts of your organization. Now say you’ve got a wiki page in one wiki that you’d rather place in a different one, or that you’d like to copy. Sure, you can edit the first wiki’s page, copy the WYSIWYG editor’s text, and paste it into a new page on the second wiki. But where’s the fun in that?

Instead, let’s move some files around to get a sense for how the Apple WikiServer stores its data.

First, recall that the Apple WikiServer stores its content inside the /Library/Collaboration directory on the computer hosting the wiki web site. Inside of this directory you’ll find four items:

$ cd /Library/Collaboration/
$ ls -la
total 8
drwxrwxr-x   6 _teamsserver  _teamsserver   204 Mar  2 13:50 .
drwxrwxr-t@ 60 root          admin         2040 Mar  3 19:16 ..
drwxr-xr-x   2 _teamsserver  _teamsserver    68 Mar  2 13:50 ArchivedGroups
drwxr-xr-x   6 _teamsserver  _teamsserver   204 Apr  4 18:48 Groups
drwxr-xr-x   2 _teamsserver  _teamsserver    68 Mar  2 13:50 Users
-rw-r--r--   1 _teamsserver  _teamsserver   249 Mar  2 13:50 dataVersion.plist

First, note that all of these files are owned solely by the _teamserver user, and only that user can access these files in any way. This keeps people like you and me from accidentally tampering with the Apple WikiServer data store. However, for the purposes of this exercise, we’re going to tamper anyway. To do so, you’ll need root permissions, which comes with a big caveat: be careful whenever you are root—one typo could erase your hard drive (and there is no undo)!

Become root by saying:

sudo su -

and supplying your password. Once you are root, you can explore further into the depths of the Apple WikiServer.

Inside the Apple WikiServer data store for wiki pages

The Apple WikiServer, which runs under the username _teamserver, is really just a bunch of organized flat files, supplemented with a number of SQLite databases (also flat files, really), and glued together with a bunch of Python scripts running under the Twisted networking framework. We’re not going to cover Python or Twisted in this post, so for further information check out their respective homepages. There’s also a great introduction to Twisted at ONLAMP.com.

To make the wiki go, the WikiServer runs these scripts to read and write data to these flat files. Let’s dig into this data store. Since the wikis are each associated with an existing group, all of the wiki files are inside the Groups subdirectory. In this directory, you’ll find one directory for each group that has a wiki. Each group directory, in turn, contains 6 other directories, 2 regular files, and 1 SQLite database:

# pwd
/Library/Collaboration/Groups/chiefs
# ls -p
discussion/	index.db	metadata.plist	resources/	wiki/
extrainfo	mailinglist/	public/		weblog/

The two regular files are metadata.plist and extrainfo. The metadata.plist file maintains the preferences of the individual wiki web site that you set from the “Group Settings” page on the wiki web site itself. That is, these settings are all editable via the web directly at http://your-server.local/groups/your-group-name/settings/. That page is also accessible to an admin user from the wiki home page under the “Admin functions” header in the sidebar of the web page (the key names in the plist file are pretty self-explanatory and map easily to the GUI options).

At the moment, the extrainfo file is a bit of mystery. It stores single lines of plain ASCII text in the form:

groups/your-group-name/web-service/page-id

where your-group-name is the name of the associated group for the file, web-service is one of wiki, weblog, and I assume can also be mailinglist or discussion, although I’ve only seen the first two on my server in practice.

What we’re most interested in for the purposes of wiki pages is, unsurprisingly, the wiki directory. This directory stores as many subdirectories as there are pages on your wiki. Every time you create a new page, a new directory is added here. If you delete that page, however, this directory is not deleted outright.

Each directory is named with the unique page identifier that the Apple WikiServer automatically generates followed by a .page extension. The exception is the welcome.page directory, special in that it contains the content of the wiki’s main (front) page. The Mac OS X Server Web Technologies Administration for Version 10.5 Leopard manual explains the contents of this directory as follows:

  • /Library/Collaboration/Groups/groupname/wiki/pagename.page/ contains the component files of a wiki page.
  • /Library/Collaboration/Groups/groupname/wiki/pagename.page/page.html contains the main text of the wiki (html content).
  • /Library/Collaboration/Groups/groupname/wiki/pagename.page/page.plist contains the metadata for the wiki page.
  • /Library/Collaboration/Groups/groupname/wiki/pagename.page/revisions.db contains the version history database for that wiki page.
  • /Library/Collaboration/Groups/groupname/pagename.page/images/ contains the images for that wiki page.
  • /Library/Collaboration/Groups/groupname/pagename.page/attachments/ contains all attachments for that wiki page.

Here are some additional details:

  • The “page.html” file contains all of the (X)HTML that appears inside the WYSIWYG editor field when you click the “Switch to HTML View” button (the arrow brackets). Nothing more, and nothing less.
  • The “page.plist” file maintains the metadata that associates a single wiki page to its group as far as the Apple WikiServer is concerned. If you take a closer look at it, you’ll find property list keys such as commentUID and uid whose values are both strings—URLs, in fact—that the Apple WikiServer uses to, for example, call the appropriate wiki theme to render around the page content. This file also stores self-explanatory data such as author, createdDate, page kind, a last modifiedDate and lastModifiedAuthor and, of course, the page title. The most interesting key in this file is the versioned key, which is a boolean that can turn versioning on or off. By default, wiki pages have this key set to true and (for example) weblog pages have this set to false. However, you can manually turn off versioning for a specific wiki page by changing this key yourself. It won’t delete past revisions, it will just stop new revisions from being saved. This is especially cool, because it means you can version-control weblog pages simply by editing the page’s associated plist file.
  • The revisions.db is the SQLite database in which all revisions are stored. The database contains a single table, revisions, that looks like this:
    CREATE TABLE revisions(revision INTEGER PRIMARY KEY AUTOINCREMENT, editType, comment, lastModifiedBy, content, time DEFAULT CURRENT_TIMESTAMP);

    The interesting bit is the content field, which itself is just a string—but a string stored as an Apple property list, in full. Some keys in the property list inside the SQLite database are the same as those inside the page.plist file for the page, but with some differences. For instance, there is no versioned key in the plist in the database (for obvious reasons), and there is a tags key (which is missing from the filesystem equivalent). Most importantly, there is a content key, which stores the content of the page.html file.

  • The revisions.db file is created automatically only if the versioned key in the page’s page.plist file is set to true.

Finally: move a wiki page from one wiki to another

Now that we understand some of the internals of the Apple WikiServer data store, let’s mess with it just a little by moving that one test wiki page from one wiki into another. Since almost everything is flat files, it’s remarkably easy to do:

# mv /Library/Collaboration/Groups/your-group-name/wiki/page-id.page /Library/Collaboration/Groups/new-group-name/

Be certain that there isn’t already an existing page in the new-group-name wiki with the same page-id as the one you’re moving. So far, Apple WikiServer appears to generate unique page IDs across the entire system (as opposed to just per-group), but I haven’t verified this yet.

With the wiki page files physically moved into the new wiki, you can now access the page at the new wiki URL. Trying to access it in the old URL will result in an HTTP 404 “Not Found” error. However, the page at the new wiki URL has the same theme as the other, old wiki. To correct this, you need to edit the page.plist file in two places: the uid string value and the commentUID string value relative URLs need their your-group-name changed to the new-group-name, whatever it is.

After you save this this change, simply reload the wiki page in your browser. Voila. Page moved.

 

Written by Meitar

April 8th, 2008 at 4:12 am

One minute Mac tip: Restore Bonjour’s “.local” addresses

leave a comment

Lately, there have been a string of networking problems with Mac OS X 10.5 Leopard reported by sites such as MacFixIt. One of most common symptoms is the loss of Bonjour’s “.local” addresses. So, for instance, if you have a machine named “Perseus” then you could address that machine by the hostname “Perseus.local” instead of its IP address.

However, if you find that the .local host name no longer works but the IP address still does, the problem may be in a corrupt or outdated local DNS cache. Luckily, the solution is incredibly simple. Just run:

dscacheutil -flushcache

at a Terminal prompt, and try again. If you’re still running Mac OS X 10.4 Tiger and you experience this problem, the solution is just as simple. Instead of the above line, simply run:

lookupd -flushcache

at a Terminal prompt.

Written by Meitar

April 5th, 2008 at 5:17 am

A web developer’s introduction to the Apple WikiServer (part 1)

10 comments

I absolutely love wikis, so when Apple introduced Mac OS X Server 10.5 “Leopard,” one of the new features I was really excited about was “WikiServer” (what the Apple marketing department calls “Teams”). I’m calling this specifically the Apple WikiServer in order to avoid confusion with the pre-existing wiki plus web server package called WikiServer.

Apple WikiServer: Mac OS X Server’s built-in Intranet builder

At work, I’m finally getting the opportunity to try the Apple WikiServer out. Its strongest asset, by far, is the integration it has with Apple’s Mac OS X permission scheme. Apple WikiServer makes heavy use of the OS‘s built-in user accounts to define users and groups, and the permissions those users and groups have to edit, view, and comment on pages in the wiki you create with it. And, because Leopard Server has full support for Access Control Lists, those permissions schemes can be as complex as you like.

This is very important because many large (and small) organizations have sensitive material that they’d like to keep private, or restricted to certain groups. Historically, wikis are a free-for-all. Anyone and everyone who has access to any part of the wiki can change any other part of the wiki. Recent wiki implementations such as later version of MediaWiki and some other wiki software have implemented permissions systems to allow administrative users to control access rights, but these are often complicated or require code-level configurations.

With Apple’s WikiServer, all of these permissions can be managed via the Workgroup Manager application, and because you can take advantage of the built-in ACL support, you can model your organizations permissions scheme directly in the Server OS permissions structure, giving you a much easier way to control information access. Take note, however, that like almost all other things that have to do with your Apache configurations, your Server’s Web Service will likely need to be stopped and started again for any changes you make to a wiki’s permissions take effect.

The Workgroup Manager application is also where you go to create new wikis for groups. To enable a wiki, you need to already have created a group and assigned users to that group. For instance, I created a Developer Wiki where all of the in-house developers can share tech tips, so I created a group called “Developers” and assigned individual developers, as well as the company executives (by way of the “Executives” group) to that group. The group-within-a-group technique is key, because if the company executives change, the members of the Developers group does not need to change, too. In all of Apple’s publications, Apple refers to the wikis hosted by WikiServer as an “intranet website.”

It’s clear that Apple intended this product for use within small companies, and not necessarily out on the open Internet. What follows are just a few notes I’ve compiled about how the Apple WikiServer works.

Front-end Code Generation from the Apple WikiServer WYSIWYG Editor

The Apple wikis are very nice to use. Their functionality is relatively straightforward to find and activate. However, the HTML code that the Apple wikis generate can be a little confusing. By default, new page text is entered into a semantically meaningless <div> element. This can be changed by highlighting text and then selecting “Paragraph” from the formatting toolbar. Subsequent paragraphs that are typed seem to then use <p> elements. However, some paragraphs revert back to <div>s when I used it, and I’m still not sure why or when this occurred.

On the plus side, so far, all the browsers I’ve used with the Apple WikiServer function the same way. This include Firefox 2, Safari 3.1, and Internet Explorer 6 and 7.

Typing actual code and having it marked up as such can’t be done in the GUI formatting toolbar to select a <code> element. The “Monospace” item in the text formatting toolbar creates <pre> elements and <pre> elements only. However, Apple does provide a “Switch to HTML view” button (the arrow brackets button) and one can enter standard HTML, including <code>…</code> elements in that view, and then switch back. This behaves perfectly on all browsers except Internet Explorer, in which your text area field shows no line breaks whatsoever.

Apple’s WYSIWYG editor handles escaping special characters when those special characters have HTML entity reference equivalents, such as double quotes (“), arrow brackets (< and >), and ampersands (&). It does not seem to handle Unicode characters, such as the ellipses in the prior paragraph. However, such Unicode characters need not be escaped as long as the document’s character set is UTF-8 (or UTF-16), which the Apple WikiServer specifies and supports out of the box.

Pressing the Return key twice causes the Apple wiki to generate an empty <div> or <p> element with an explicit break (<br />) inside of it. One can deduce that this is a design choice in order to help transition users who are used to plain <textarea /> inputs to Apple’s WYSIWYG editor. It’s also the only way to space paragraphs properly if the user hasn’t selected the “Paragraph” option in the text formatting toolbar. Otherwise, simply hitting the Return key once is enough to space paragraphs apart properly (i.e., the functionality is equivalent to the way Microsoft Word or Pages handles paragraph breaks).

Interestingly, the “Enter URL…” functionality form the toolbar is smarter than one might first assume. For instance, it recognizes email address and prepends a mailto: scheme to the link if it finds one. This is contrary to what the Apple-provided manual states, which tells you to enter the “mailto:” portion as part of the URL. In fact, you should omit this, else your final mailto link will actually read “mailto:mailto:your.email@address.com”.

This means linking to “mailto: links” is as simple as typing an email address. Similarly, the WYSIWYG doesn’t complain if your fully-qualified URL doesn’t include a scheme, so you can enter //apple.com/ and the subsequent link is generated as <a href="//apple.com/">Link text</a>. This is one step above and beyond even WordPress’s new WYSIWYG editor, which forcefully prepends an http: scheme to URLs without one.

For the most part, copy-and-paste works as expected, except in cases where the WYSIWYG editor does not understand the current formatting, such as a specific font and (and this is a biggie) for links. At first, the editor will appear to show that the formatting (including links) is saved, but when you actually save the page, only the formatting that the WYSIWYG editor understands is actually saved. Worse, all your links are turned into underlined—but unlinked—text. In short, this means that if you are copying and pasting page content that contains links, you need to do so in the HTML view of the page editor.

Inexplicably, the editor generates <i> and <b> tags for italics and bold, instead of the preferred <em> and <strong> elements. I’m not sure I understand why this is the case. There does exist a an “emphasis” option in the toolbar, as does an “important” option, but these generate strange spans instead. The “Important” item wraps the selected text inside a <span class=”Apple-style-span custom_forecolor_important”>…</span> and the “Emphasis” item wraps the selected text inside a <span class=”Apple-style-span custom_forecolor_emphasis”>…</span> element.

There’s also one other item, “Highlight,” which wraps the selected text inside of a <span class=”Apple-style-span custom_backcolor_highlight”>…</span> element.

The only explanation that makes sense to me, after much speculation, is that perhaps Apple does not want to encode semantic information such as what <em> and <strong> would imply from users who use a WYSIWYG editor. This shows either a blatant distrust of users or incredible foresight. I’m not sure which.

Page names and URLs

Currently, the search functionality built into Apple’s WikiServer only searches on the text of a page’s title.

Apple’s WikiServer generates unique page names for each new wiki page. These names consist of two parts, but only one seems to make any difference. For example, if you create a new Wiki page called “Hello”, you might get an address in your web browser’s location bar that looks like this:

http://your-server.local/groups/your-group-name/wiki/1d06a/Hello.html

Most of this is standard URL stuff (the protocol, the server address), and most of the rest is self-explanatory (the group name, the wiki section). The important bits in this URL are the last two:

  • 1d06a
  • Hello.html

Obviously, “Hello.html” came from the fact that you named your new page “Hello”. WikiServer appended “.html” on its own. The other bit, the short string of random characters, is a unique identifier used across all of this group’s web services (the wiki, blog, calendar, and mail list, if these other services are enabled) to uniquely identify this page. What’s interesting about this is that only the unique string of characters seems to matter in regards to accessing the page. That is, if you next ask for:

http://your-server.local/groups/your-group-name/wiki/1d06a/Some-Random-Page.html

you’ll still get the same “Hello.html” page from before, even though you’re seemingly asking for “Some-Random-Page.html”. In fact, it doesn’t seem to matter what you replace “Some-Random-Page” with. So long as there is some text in that part of the URL, that the URL ends with “.html” and that the unique identifier remains untouched, you’ll always end up retrieving the “Hello.html” page.

This means that if you change this page’s name later to, for instance, “Hello world”, old links that point to “…/1d06a/Hello.html” will continue to work, even while new links will start to point at “…/1d06a/Hello_world.html”. From a usability perspective, this is simple and effective; it ensures that users have a reminder of what the page is about by looking at the last part of the URL. However, once page names change, it becomes a bit non-optimal, because the same page can be referred to by multiple names—a “no-no” in the SEO world and a practice discouraged by most semantic-web types.

I would imagine that Apple made this design decision because the company envisioned their WikiServer to be used, again, primarily in intranet and SOHO environments, and as a result are not too concerned with search engine optimization. As an aside, this unique string is also how Apple’s WikiServer identifies the stored content on the filesystem. Read on for more details.

Hacking the Apple WikiServer

There isn’t a lot of information out on the Web right now about how to work with the WikiServer, especially for developers. Therefore, some digging is needed. After a bit of research, I discovered the following key directories that the WikiServer uses. They are as follows:

  • /usr/share/collaboration - This has a few developer tool support files as well as the majority of the client-side code for the Wiki (javascripts, etc).
  • /usr/share/wikid - This directory holds the Python sources and compiled bytecode for all the “Teams” components (including wiki, blog, calendar, etc.). It seems to run on Twisted and a number of other familiar-sounding components.
  • /Library/Application Support/Apple/WikiServer - This is where most the data is stored, inside of plist files and a few others. The Themes subdirectory here is where Apple recommends that look-and-feel changes be made.
  • /Library/Collaboration – This is the default data storage location for all the “Teams” components. The actual content of the wikis and blogs will be kept somewhere in this directory, which means that this is the directory you want to backup to backup the content of your wikis. This location is the only user-configurable one of the bunch. To change it, change the “Data Store” value in Server Admin. (A more detailed listing of this directory hierarchy is available on page 62 of the Mac OS X Server Web Technologies Administration For Version 10.5 Leopard manual.)

If you take a peek at the /Library/Collaboration directory, and follow that into the Groups/your-group-name/wiki directory, you’ll find a list of all the pages in your wiki stored as .page bundles, identified with the unique character string WikiServer generated when it first created the page.

It should be noted that anything in the /usr/share directory will likely be overridden whenever Apple releases an update that modifies the WikiServer. As a result, any and all changes you make to WikiServer’s templates or themes should be done by creating new files in the /Library/Application Support/Apple/WikiServer/Themes directory.

It’s interesting to note that the WikiServer seems to use Python for its back-end processing. This may open up some interesting integration possibilities for Python programmers in the future.

More help eslewhere

Even though Apple WikiServer is relatively new, there’s a load of helpful information about it on the web. Most of the good stuff is on Apple’s own Discussions boards, but more and more info is beginning to show up on blog posts. A Google search should give you what you need. For the really lazy, however, here are a few helpful items:

This was just a brief introduction to WikiServer from some notes I’ve been collecting in my experimentations, but I hope it’s helpful to someone somewhere. Cheers. Or, continue to Part 2.

Written by Meitar

April 5th, 2008 at 5:10 am