Ups and Downs and Everything In Between

For those who have been wondering what is keeping me so busy these days, the answer is that I’m working on the final stages of a book that is getting published as one of three co-authors. Not only am contributing three chapters (the technical chapters on (X)HTML and CSS, specifically), but I am also technically reviewing the entire book.

My co-authors on the book, called Foundation Web Standards Foundation Website Creation (you can pre-order now) and published by Friends of ED, an Apress company, are Jonathan Lane of Industry Interactive, Inc. and Joe Lewis, who blogs at Sanbeiji.com. I’m not going to say much more until after the book is released in late July.

For the eager, here’s the description of the book posted on the Friends of ED website:

Foundation Website Creation explores the process of constructing a web site from start to finish. There is more to the process than just knowing HTML! Designers and developers must follow a proper process to flush out goals and objectives and determine requirements both prior to, and during project development.

Large Web projects are rarely completed by a single person. Producers, project managers, designers, developers, writers, and editors all play critical parts in a project’s evolution. This book provides an overview of the entire process, and also shows project development from the perspective of these different roles. It introduces the key concepts and duties performed by every member of such a team, and gives you the skills necessary to tackle projects like a professional.

It’s quite exciting getting a book out, and it’s quite a bit more work than I’d have ever originally thought. That being said, it’s extremely rewarding. There’s a lot more work I need to do on it between now and the time it gets released to publishing, so, well…back to work I go.

Now you all know where I’ve been spending my time writing.

In a (rather beastly) project at work today, I found myself needing to import a significant number of contributed Drupal modules into Subversion vendor branches to prepare for custom development. To do so manually would have been quite the hassle, so after downloading the appropriate tarballs and creating a module_name/current directory under my vendor/drupal/modules vendor branch directory, I concocted this little (relatively untested) script to handle the mass tagging operations I needed to perform.

for i in *; do
    v=`grep 'version = "' "$i/current/$i/"*.info |
      cut -d ':' -f 2 |
        sed -e 's/^version = "/v/' -e 's/"$//'`
    svn cp "$i/current" "$i/$v"
done;

It’s a bit buggy for some modules that have multiple .info files, but I’m sure a few more pipeline stages can fix that. (Which, because I’m done with this at the moment, I will leave as an exercise to the reader.)

Chalk this one up as another testament to the power of shell scripting and how it can help every developer get their job done faster.

Using the pbpaste and pbcopy commands, you can manipulate the contents of the Mac OS X clipboard (or more formally known as the pasteboard) right from the command line. As a brief example, just select the text of this first paragraph, copy it to your clipboard (with -c), and then type pbpaste in a Terminal prompt. You should see output similar to the following:

Perseus:~ meitar$ pbpaste
Using the pbpaste and pbcopy commands, you can manipulate the contents of the Mac OS X clipboard (or more formally known as the pasteboard) right from the command line. As a brief example, just select the text of this first paragraph, copy it to your clipboard (with ?-c), and then type pbpaste in a Terminal prompt. You should see output similar to the following:Perseus:~ meitar$ 

Pretty straightforward, right? The only thing to be aware of is that the  symbol showed up in the output as a ? symbol. This is because the Terminal doesn’t support Unicode, but that’s a topic for another time.

Anyway, what’s happening here is nothing more magical than simply reading the clipboard and pasting it into a command’s standard output stream. As a result, you can construct pipelines that read from or add content to the clipboard. Here’s an example in reverse, which takes a command’s standard output and replaces the contents of the clipboard with it:

echo "Hello world! I came from the command line, but now I'm in the clipboard." | pbcopy

This command produces no output, but if we examine the contents of the clipboard (by selecting Edit → Show Clipboard from the Finder’s menu bar) we can see that the text we echoed has indeed been copied there.

Another way we can verify that this worked as expected is to simply pbpaste again:

Perseus:~ meitar$ pbpaste
Hello world! I came from the command line, but now I'm in the clipboard.
Perseus:~ meitar$ 

Anyway, this is cool, but it isn’t very useful yet. For that, we need to add some more stages to our pipeline. Let’s take the simple case of trying to count how many words the selected text contains. We’ll use the clipboard contents shown above to do this:

Perseus:~ meitar$ pbpaste | wc -w
      14

Using the word count (wc) utility, we can count words (-w) very easily. Indeed, our previous example does have exactly 14 words in it. We can also count characters (-c) or lines (-l) of text in the clipboard this way. This is like adding Microsoft Word’s “Word Count” feature to every single piece of text you can copy!

As another example take, for instance, the simple case of copying and pasting a snippet of email from Mail. Instead of pasting it back into a text file verbatim, let’s prepend ‘> ‘ to the beginning of each line. This way, when we paste our email’s snippet, we’ll know where the snippet begins and where it ends. This is a simple three-stage pipeline that uses pbpaste to take our clipboard and put it into the pipeline and then reads back the result from the pipeline back to the clipboard using pbcopy. In the middle, we use sed to insert the desired text at the start of each line:

pbpaste | sed -e 's/^/> /' | pbcopy

Now, when you paste your clipboard, you’ll have a greater-than symbol at the start of each line. Naturally, check out the manual pages for all of these commands for more detailed information. For instance, type man pbpaste for more information about the pbpaste command.

Since Mac OS X 10.2 Jaguar, Mac users have been accustomed to the ease of use of Apple’s very cool Keychain Services technology. The Mac OS X Keychain basically a secure database of all your passwords, sorted into files called (unsurprisingly enough) “keychains.” Each user account on a Mac OS X system has a login.keychain, and the system itself also has a system.keychain.

Whenever you tell an application to “Remember this password in my keychain,” what you’re doing is writing a new encrypted entry into your user account’s ~/Library/Keychains/login.keychain file. Then, the next time the application needs to access a restricted resource, it just asks Mac OS X to get the password for it. Of course, all of this happens automatically, so except for that single checkbox most users probably don’t know that the keychain even exists.

What’s even more awesome than all of this automagic password storing action, though, is the fact that Apple has also provided an easy-to-use application to manipulate the keychain yourself. What good does this do us? Plenty! Observe.

Say you’ve just signed up with a new ISP. They send you a username and a password to log on to their ADSL network with. Of course, they send this password to you on paper—how insecure! Instead, after changing the password to something else first (something other than mypassword, which is the example password I’ll use here), we can use Mac OS X Keychain to securely store the password and retrieve it later.

1. First, launch the Keychain Access application located in the /Applications/Utilities folder of your startup drive.
2. Next, click the “Create a new Keychain item” button (the +) button near the lower left-hand corner of the window. The Add Keychain Item sheet appears.
3. Enter a meaningful name, such as “ADSL ISP Account” in my example, in the Keychain Item Name field.
4. Enter the username or account name associated with this password in the Account Name field.
5. Enter the password into the Password field.
6. Click the Add button.

That’s all there is to it. To later retrieve your password if, say, you ever forget it:

1. Launch the Keychain Access application.
2. Locate and double-click the keychain item that stores the account and password information you want to retrieve.
3. Tick the “Show password” checkbox. You’ll be presented with a dialogue box that asks for your keychain’s master password. Unless you’ve already set it to something else, this is the same password you use to log in to your Mac OS X user account.
4. Enter your keychain password and click “Allow.” If you click “Always Allow” instead, Keychain Access will not prompt you for your login keychain’s password the next time you ask to see this particular password. I never press that button.
5. Your password’s plaintext is now visible.

This effectively obviates the need for third-party applications such as Password Gorilla, PasswordWallet or KeePassX which are great programs, but all suffer from a lack of a good user interface. Furthermore, there’s no reason why we can’t store short arbitrary strings of sensitive information in the keychain temporarily. Sure, it might clutter up your keychain, but you can always search the entries using the standard Mac OS X filter search bar at the top right of the window.

In fact, Apple’s been kind enough to offer an interface to do just that in an even more effective way, called Secure Notes. These are simply plain text strings of arbitrary length that can be stored securely inside your keychain, and that use the same interface to access (requiring your password to view). The only real difference is that instead of a single line, you’re given a fully scrollable text area in which to type your secure note.

Moreover, because keychains can be synced to multiple Macs with .Mac Sync (or a third-party synchronization solution), you can always have access to all your passwords regardless of which physical Mac you’re using. Best of all, since you never have to remember another password ever again, you can quit using the same password for multiple accounts, and you can always use really hard-to-crack passwords.

I subscribe to a number of really great technology newsletters because they interest me. One of these is the XML.com weekly newsletter. XML is a technology that has exploded in the last several years, and its specifically an area that I, as a front-end and semantic web specialist, find exceptionally intriguing. Most intriguing today, however, was Managing Editor of XML.com Kurt Cagle’s article titled Is Telecommuting the Next Wave for Education? in the XML.com newsletter.

Not only was it a great article that highlights a particular XML schema for education-related material produced by the Schools Interoperability Framework Association, it paints the picture of a future I’ve already lived ten years ago. I am heartened to see that, at long last, other people are beginning to see the benefits of technology used for “distance learning.” Frankly, I can’t imagine learning any other way.

I tried to find this article online so I could link to it, but I could not. Instead, I’m reposting the article in full here. If this is available somewhere on the ‘net that I just haven’t found yet, please let me know.

Is Telecommuting the Next Wave for Education? by Kurt Cagle, Managing Editor,xml.com.

In the great analysis game, I have two particularly adept spies - my daughters. Over the years I’ve noticed that both of my daughters tend to be remarkably good barometers about the way that the wind is blowing with regard to youth trends, which in turn tend to be significant because teenagers in particular often tend to be the earliest adopters of new technologies. If something doesn’t resonate with them, no matter how big a marketing budget, it usually doesn’t fly.

One of the things I’ve noticed with my eldest daughter in particular is how many of her friends either are or have been recently “home-schooled”. Now, for many, home-schooling has long been associated with religious organizations, particularly evangelical Christians, who feel that the school system is too secular for raising their kids. However, given that the kids I know (and their parents) are generally not in that community, I was a little puzzled with what was going on.

Instead, what seems to be happening is a phenomenon that I think will have major ramifications for society, and certainly for the tech community. As the Internet was taking off around 2000, a lot of school districts began implementing a program to help those people who were often at significant distances from a school by offering certain classes online - with exercises online, video conferencing and periodic tests. At first, these classes were ones that you would expect to make the migration - science and math courses - but over time, they have extended to cover everything except those classes that require group participation - band, or choir, for example - or need physical facilities, such as wood-working. Physical education requirements could generally be met by agreeing to some form of monitored activity - swimming classes at a recreational center, for instance.

Yet a funny thing happened while setting up these distance learning programs. While remote users became enthusiastic participants in this new wave, the largest group of users have been urban or suburban kids who, for one reason or another, didn’t fit well within the school paradigm. In some cases, the people who took advantage of these courses were students who were involved in focused activities that involved travel - talented musicians who were often on the road, dancers, athletes who were often involved in activities at different schools or other events, or those whose family travelled frequently, who took to the distance courses because they were able to learn better around their other activities. In some cases, the students were people who were going through emotional issues at home - a death or divorce, for instance, though increasingly that has made its way up into those students who just couldn’t face the high-pressure world of middle-school or high-school dynamics.

In other words, the kids and teenagers who were taking advantage of these courses just recognized that it gave them an opportunity to learn in the same way that they are increasingly interacting with the rest of the world - through the computer. Teenagers are hard wired to be more alert in the evening, and typically to be sluggish in the morning, yet school as it is set up right now forces them to be capable of handling complex math and science first thing in the day, when they are generally least reponsive to learning much of anything. Then they are forced to take home dozens of pages of homework that will force them to stay up until late in the evening anyway, meaning that by the time they reach the end of the week, they are physically and mentally exhausted.

Distance Learning lets them combine the homework with the schoolwork, so they can practice new concepts when they’re presented, not after the concepts have become hazy after a full day. It moves them away from the tyranny of the timetable so that if they are having trouble learning a concept, they can spend the time they need to master it, rather than stopping abruptly halfway through because they have to move onto their next class. It also means that if they have mastered a concept, they do not have to sit around bored while others are still trying to figure out something.

It also lets them have access to the rest of the Internet to use as a research tool. While traditional academics may shudder at this notion, as a recent controversy at one university showcased when a student was expelled for setting up a study group on Facebook, the reality is that we’re moving past the point at which we need to keep a vast storehouse of information locked up in within our brains. In an era of information ubiquity, many of the skills that are taught in schools are beginning to seem increasingly quaint, and the teachers that are effective are typically the ones that have managed to incorporate this info-sphere in their own teachings. One social studies teacher of my acquaintance in particular has become quite effective at teaching using PowerPoint, Wikipedia, and other multimedia resources, and he goes out of his way to teach children not the history itself (which they can generally look up) but how to research and analyze that history and take from it any lessons that a given period may have to offer.

Distance Learning programs are also becoming more popular for the same reason that telecommuting is becoming popular - school districts are facing increasing prices for gas and food as a typical family is, but multiplied by several thousand. Many school districts are responding to this by cutting down on the routes that their school buses follow (or in some cases eliminating bus systems entirely) forcing parents to take their kids themselves, often, ironically, increasing the total gas use dramatically and certainly causing headaches for parents who have to integrate their kids into their own commuting schedules. Similarly, school cafeteria programs are being scaled back or eliminated entirely because the cost of the food is becoming painfully high. Add into that aging infrastructures for schools, in a time when the population itself is aging (and hence less likely to fund school initiatives if it affects property taxes), and what you have is a recipe for disintegrating school systems.

Given that, the idea of distance learning as an alternative is one that may be popular at the school board level, as in general, you can generally buy a whole lot of educational training and assistance for what it normally costs to move and feed kids. While it does require some retraining on the part of the teachers, they’re also attracted to it because their job frequently involves trying to keep order over thirty to as many as forty kids, many of whom simply do not want to be there - and the same time-shifting that occurs for the students typically occurs for them as well. Indeed, in many cases what ends up happening is that the teachers pre-record certain segments of their lectures (and are increasingly posting them up on YouTube), then set up one-on-one or group chat sessions with the students.

As for the parents in this equation, I’ve often suspected that one of the reasons for the decline in the quality of contemporary education has been the fact that school has become the place where parents warehouse their kids for the day. Home-schooling does require more parental involvement, but in general it is also far easier for those parents to keep a fairly close eye on what their kids are up to in this day and age when the parents are not at home, at least for kids of a certain age. What’s more, schools are in many ways like hospitals - while their ostensible purpose is education, the chance of their kids learning bad habits is far higher in school settings than they generally are at home - and the kids that are trying to be good aren’t distracted by the ones determined to make trouble.

That’s not to say that there aren’t distractions. My eldest daughter has been known to keep a chat window up talking to friends when she should be concentrating on homework, though in fairness to her, what she’s chatting about often involves that same homework, albeit in a rather disconnected fashion:

Kat: I'm :-( ):-\. Zuko shippeded Kataara. Urgh ..
Neechan: Urrgghhhhh.
Hey, what's I if volts is 10 and ohms = 3 % (can't write the squiggle for
ohms)?
Kat: Um V =IR so, uhm, I is RV ...
Neechan: No, R over V.
Kat: Oh. Right. :-) .... so 3 divided by 10 - .3. Bad Zuko! ...

Is this bad? No - it’s only different from the way that those of us who grew up pre-Internet see the world. Distance Learning does not change the amount of homework (though I suspect that it cuts down pretty dramatically on the makework that tends to deaden interest in a subject rather than reinforcing the concepts), nor does it change the need for accountability. The kids still need to test, and still need to show that they have learned, but I suspect that their retention rates will likely be considerably higher if they can learn in a way that works well for them.

Long term, I think that this will likely end up deconstructing the traditional school system, though this is a process that will take decades to happen completely. It means, for the aggressive learners, that they could in fact complete a formal curriculum in a fraction of the time necessary, though a good teacher can work with those particular students to provide additional areas of study for them to engage with. It means that slower students can learn at their own pace, and can generally be flagged for additional help if they fall sufficiently behind. Money that school districts save in terms of providing physical infrastructure in transportation can be spent on those activities that do promote socialization - music, art, theatre, sports, civic days, and so forth - and monthly activities that bring students together can keep the bond in place of school as community.

Distributed education is part of the larger process of social redistribution that is occurring because of the Internet. Modern education emerged about the same time that the modern corporation emerged - in the 1930s - during an era when the dominant forces at play involved hierarchies, centralization, consolidation and economies of scale. In this decade, the dominant forces are network related - decentralization, the economies of global localities, the disintermediation of authority and the a shift away from the geographical. Just as these forces are resisted at the corporate level despite the obvious benefits (and just as workers in places that can telecommute are increasingly doing so) , these same forces are resisted at the educational level with much the same results - students (and their parents) are taking advantage of any loophole they can to make it available to their kids because it results in a better education for them).

So far, most educational software is boutiqueware, typically Flash-like applications. This won’t necessarily change moving forward (it is hard in general to make educational software that doesn’t have a strong “games-like” component, and for the most part that game approach makes the educational software far more engaging than it would be otherwise), but one thing that will need to happen as the distributed systems move forward is for the emergence of some kind of general framework for the exchange of educational related information. One particularly promising start in this direction is the development of the Schools Interoperability Framework (SIF) (http://www.sifinfo.org/sif-specification.asp), which is an XML standard most recently updated in March 2008. Its mandate is fairly broad - providing XML schemas for describing grade-books, library services, student information, instructional services and so forth - (see Figure 1. SIF Zone Services).Figure 1. SIF Zone Services.

One of the jobs that the XML community needs to do is to reach out to the developers of educational software and insure that they are aware of the SIF standards, in order to provide better interoperability between their core applications and the growing educational educational noosphere, and to reach out to educators and education IT departments (which are all too typically the math teacher in his spare time) to make them cognizant of these same standards and to help implement solutions around these standards. As more students opt to go “virtual” the ability to maintain consistent, and more important interoperable, records becomes ever more paramount.

Long term, the move towards distributed education will shape society in some very profound ways. The kids going to school in such an environment today are more adept at the art of self-education, are usually more capable at analysis and research, and because they managed to avoid the often harsh emotional trauma of dealing with several hundred other kids of the same age daily, usually have more self-confidence than their school-based peers. They’ll likely have little patience for the Tayloresque approach to college education and will continue their lives in a similar manner by educating themselves within the college’s online environment (and will tend to shun those colleges that don’t offer such services) and when they start coming into the workforce in sufficient numbers, they will reshape the way that organizations are set up. On the flip side, I think this is likely to cause a huge amount of cultural friction between this generation and those educated more traditionally in previous generations, because their respective realities will be very different.

So far, the movement is still just a trickle, but watch this space closely - it will become a flood soon enough - and the bricks in the wall will come tumbling down (to paraphrase Pink Floyd).

As I’ve been blogging about, I’ve been playing a lot with Apple’s new WikiServer (or “Teams Server”) at work. We’re still evaluating what we’d like to use it for, but as part of the experiments, I’ve been finding myself having to do some pretty crazy things with the WikiServer. This one is pretty bizarre, and is probably not only very dangerous for the content of your group’s wiki and blog, but also almost certainly not a best-practice.

With that caveat out of the way, here’s how I managed to move all the pages in an Apple WikiServer group wiki and blog from one group to another.

Dear God, please have a backup!

Okay, step 1 is mundane, but seriously, please of please have a backup of your data before you do any of these things. To make a back up of your entire WikiServer’s data store, simply:

sudo tar -cvzf backup-file-name.tgz /Library/Collaboration

If anything goes wrong, you can restore from your backup just as simply:

sudo tar -C /Library/Collaboration -xvzf backup-file-name.tgz

Okay, with that out of the way, next make sure absolutely nobody is using any of the Group services for the group you are going to perform the move from, or to. People can work on other group’s wikis, it’s just the ones you’ll be touching you want people to avoid. You can enforce this with some Apache redirects, which is left as an exercise to the reader.

Step 1: Rename or create a new Group in Workgroup Manager

First, you need to either rename or create a new group in Workgroup Manager. If you make a new group, be sure to enable all the services that the old group used.

If you’re simply renaming a group, then you might not even have to go through this trouble. You merely need to change the group’s “Name” (as opposed to its “Short Name”) and then stop and start the Web Service from Server Admin to see the group’s name change. However, if you also want to change the group’s “Short Name” (i.e., the group’s POSIX group account symbolic name), then you will need to perform these steps.

Once you have the new group ready to go in Workgroup Manager and you have stopped and started your Web Service in Server Admin, continue to the next step.

Step 2: rsync all your files from the old group to the new group

This is simple. Just run:

sudo rsync -avzE --progress /Library/Collaboration/Groups/old-group/ /Library/Collaboration/Groups/new-group/

Note that the trailing slashes on the directory names in this command are quite important, as they tell rsync to take the contents of the first directory and place those items as the contents of the second directory. Without the trailing slashes, rsync will make extraneous directories for you, which Apple WikiServer won’t understand. See man rsync for more information.

Step 3: Update the plists for all your WikiServer pages in the new group

Now, WikiServer has all the content of your old group but all of the internal references are wrong, since they still point to the old group. What you need to do is rewrite all those references so that they point to the new group address. The group references are stored in property list files. See my older blog entries for details about the filesystem structure of Apple’s WikiServer data storage layout.

To do this is a relatively simple procedure. As root, do the following, and do it carefully because as root you can easily mess up (and there is no undo button on the command line!):

sudo su -
cd /Library/Collaboration/Groups/new-group
grep -ri "old-group" * | grep '^[^B]' | cut -d ':' -f 1 | grep 'plist$' | sort | uniq > /tmp/list_of_plist_files_to_edit
for i in `cat /tmp/list_of_plist_files_to_edit`; do sed -e 's/<string>groups\/old-group/<string>groups\/new-group/' $i > $i.new; mv "$i.new" "$i"; chown teamsserver:teamsserver "$i"; chmod o-rwx "$i" done;

In English, this means:

1. Become root.
2. Change to the /Library/Collaboration/Groups/new-group directory
3. Find all plist files that have the the string old-group in them, sort them, and write this list of files to the /tmp/list_of_plist_files_to_edit file.
4. For each of the files listed in the /tmp/list_of_plist_files_to_edit, find the text string <string>groups/old-group and replace that text with <string>groups/new-group, and save this change as the name of the file with .new appended. Finally, replace the original file with the file we modified, and give them the appropriate permissions.

Step 4: Create an Apache redirect to make sure no HTML links are broken

Okay, at this point your new group is up and running and it should be working. However, if you had any links at all in any of your group’s pages, they are now all broken because they still point to the old group. Rather than going through the HTML itself and cleaning this up right now (because that’s very error-prone indeed, even with automated tools), it’s much easier to just tell Apache to redirect all requests for the old group to the new group.

To do this, edit the Apache configuration file of whatever Virtual Host you have been serving the WikiServer from. Most of the time, this will be at /etc/httpd/sites/0000_any_80_.conf.

The end of that file probably looks something like this::

#       Include /etc/httpd/httpd_users.conf
#       Include /etc/httpd/httpd_directory.conf
        Include /etc/httpd/httpd_groups.conf
        Include /etc/httpd/httpd_teams_required.conf
        LogLevel warn
        ServerAlias *
</VirtualHost>

Right above these Include directives, simply add the following:

        <Location /groups>
            <IfModule mod_alias.c>
                Redirect 301 /groups/old-group http://your.server.address/groups/new-group
            </IfModule>
        </Location>
#       Include /etc/httpd/httpd_users.conf
#       Include /etc/httpd/httpd_directory.conf
        Include /etc/httpd/httpd_groups.conf
        Include /etc/httpd/httpd_teams_required.conf
        LogLevel warn
        ServerAlias *
</VirtualHost>

There you have it. New Teams Server group, old group’s data.

Caveats

Note that this does not update any of the SQLite databases used to store things like revision history and so forth. These things are, for the most part, not really necessary to update but it would be ideal if the old plist revisions could be changed in there, too. That’s not so much more extra work, really, but I’ve found it typically unnecessary except in fringe cases, so I leave that as an exercise to the reader.

If you do this to your WikiServer and do not update the SQLite databases as well, just be mindful of that fact so that you’re not surprised if something goes wonky down the line.