The problem is that I often use the Terminal to do a variety of tasks and a lot of the commands that I use on a daily basis do not play well at all with Finder comments: for example if you use cp or mv to move files around, the affected files will not keep their Comment entry. This is because the metadata is not kept with the file (possibly using the extended attributes) but instead is stored in the .DS_Store file located in the parent directory of any given file. Neither rdiff-backup nor rsync (which I often use to transfer files to a Linux server) play well with metadata and you risk that your backup does not properly keep the Finder (or iPhoto) Comment. If instead you move files around with Finder (or use other backup solutions like Super Duper), your metadata will smoothly follow the files in their new location.

For a short while I toyed with the idea to code a cp/mv replacement, either through a bash or python script, but for the moment I decided that I would like at least to be able to create a backup (snapshot) of all the current Finder/iPhoto comments.

This is pretty easy using Spotlight’s command line utilities. If you want to get a list of the files in the $HOME dir that have a Finder Comment you can simply do:

mdfind “kMDItemFinderComment == ‘*’” -onlyin $HOME

Instead to search for comments added by other tools like iPhoto or iTunes, since those are saved within the kMDItemComment property, you can simply use the following line:

mdfind “kMDItemComment == ‘*’” -onlyin $HOME

So I cooked a small bash script to dump all this information in a file on a regular basis (i.e. when I rsync my data to the server):

DATE=`/bin/date +%Y%m%d_%H%M`
mdfind “kMDItemFinderComment == ‘*’” -onlyin $HOME > .CommentsTmp
mdfind “kMDItemComment == ‘*’” -onlyin $HOME >> .CommentsTmp
touch CommentsBackup_$DATE.comments
while read line
do
mdls -name kMDItemFinderComment -name kMDItemComment “$line” >> CommentsBackup_$DATE.comments
done < .CommentsTmp

At least now I have a way to extract this data from the store and put it somewhere in ASCII format. Next I’ll try to do a python script to ease the management of the tags and maybe avoid the situations where I inadvertently move a file with the Terminal and loose its tags. If anybody has a better idea, let me know.

The geek in me has already found some interesting content there, notably the Computer Systems Colloquium laboratory at Stanford, with an interesting lecture (available in video) from Professor Dave Patterson on the Berkeley View on Parallel Computing. On the UC Berkeley iTunes site are, among others, the audio recordings of lectures from Professor Jan Rabaey on Low Power Design (some slide material can be found on his homepage at Berkeley). For anyone interested in low-power design techniques, those are a must read!

I’m sure there are many other universities that have their own iTunes U sites. I mentioned the two I researched so far that have the most material close to my professional domain.

While reading “Inside the Machine” (see my earlier post), I thought about the times (many many years ago!) when I was doing (demo)coding on my old 486 (then substituted by an AMD K5), using nice little tricks to optimize the execution of inner loops and writing assembly code day and night. In ’97 I wrote a little doc, which is still lying around in my hard drive (and somewhere non the net), on the subject of fast texture mapping by using cache-friendly tiling techniques. Just for fun I decided to restore this doc and put it in this blog. Keep in mind it’s almost 10 years old! :-)

Read on…

Tiled Texture Mapping for pow2 Texture Sizes
Author: TheGlide/SpinningKids – June 1998

Introduction

I assume here you know the basics of texture mapping, as eplained in fatmap and fatmap2 docs by MRI/Doomsday (see on MRI’s homepage), since the present document is just an extension of those.

This doc is about doing texture mapping using texture maps stored as tiles, namely 8×8 pixels tiles. Storing the maps this way can improve very much cache access. Most of the time we have to traverse the texture through non-horizontal lines, and this causes many cache misses. The worst situation happens when we have to traverse the texture vertically: each texel we access will be on a different row and this will require, from the processor side, a whole cache line load, which is very slow.Storing the texture in 8×8 tiles ensures that every tile fits in two 32 bytes cache lines (on the Pentium), and as we traverse the texture we have a greater chance to read from the same cache line for a longer time.

Let’s assume for the moment that you have 256×256 textures.

So the U (x-axis) and V (y-axis) coordinates take up 8 bits:

U : xxxxxxxx
V : xxxxxxxx

Tiling – method 1

The first way to tile the map in 8×8 tiles is this one:

where numbers 0, 1, 2… indicate the order by which the 8×8 tiles are stored in memory (each tile being stored in 64 consecutive addresses in memory).

This way we can go from the original (U, V) coordinates to the ones in the tiled map with the following relationship:

U : xxxxxXXX -> U' = 00000xxxxx000XXX
V : xxxxxXXX -> V' = xxxxx00000XXX000

i.e. in C code:

U' = (U&0x7)|((U<<3)&0x7c0);
V' = ((V<<3)&0x38)|((V<<8)&f800);

Basically the lower 3 bits of both U and V (XXX, above) are used to address the texel inside a single tile, whereas the 5 upper bits (xxxxx, above) are used to select the tile inside the texture. The C code to convert normal texture coordinates (U,V) to tiled-texture (U’, V’) coordinates is the following:

U' = (U&0x7)|((U<<3)&0x7c0);
V' = ((V<<3)&0x38)|((V<<8)&f800);

This code enables us to convert a straight texture to a tiled texture:

tiledtmap [U'+V'] = tmap [U+V*256]

Tiling – method 2 – the better method

There’s another way to tile a texture map in 8×8 tiles. This one, which uses a different ordering of the tiles:

With this tiling method we get from the (U, V) coordinates within the original texture to (U’, V’) coordinates relative to the tiled map with this method:

u : xxxxxXXX -> u' = xxxxx00000000XXX
v : xxxxxXXX -> v' = 00000xxxxxXXX000

The corresponding C code is:

u' = (u&0x7)|((u<<8)&0xf800);
v' = (v<<3);

and as before it can be readily plugged in a converter from straight textures to tiled textures.

The code really ‘looks better’ than the first method above. It is easier and faster to convert from V to V’. That’s why we will choose this second tiling method.

Now, we could easily get our usual texture map scanline filler, put those relations inside the inner loop, and see the result. Slooow! At the expense of a little overhead, we can get a loop that is really little and optimized. So what can we do to directly use U’ and V’ in the loop and the corresponding dU’ and dV’, and read directly from the tiled texture?

We convert all of our starting U and V, and the corresponding deltas (dU, dV) that are calculated in the texture mapper before entering the inner loop (all quantities in 8.16 fixed point format, xxx is the integer part, XXX is the fractional part):

U : xxxxxxxx,XXXXXXXXXXXXXXXx -> U' = xxxxx00000000xxx,0XXXXXXXXXXXXXXX
V : xxxxxxxx,XXXXXXXXXXXXXXXx -> V' = 00000xxxxxxxx000,0XXXXXXXXXXXXXXX
dU : xxxxxxxx,XXXXXXXXXXXXXXXx -> dU' = xxxxx11111111xxx,1XXXXXXXXXXXXXXX
dV : xxxxxxxx,XXXXXXXXXXXXXXXx -> dV' = 00000xxxxxxxx111,1XXXXXXXXXXXXXXX

We have to fill the gaps in (dU’, dV’) with 1s because when we add them to the current (U’, V’) values we must propagate the carry from the lower bits to the bits that lie after the gap. After the addition we must not forget to mask out the 1s from the (U’, V’) we obtain.

Of the 16 bit fractional part we keep only the upper 15 bits. There’s a valid reason to do this: when calculating the offset to access the texel we add U’ and V’ and shift right by 16. If we kept all of the fractional bits, an hypotetical carry would propagate to the integer part, thus influencing the offset value. Keeping instead only the upper 15 bits of the fractional part, and putting a 1 bit gap between fractional and integer part the problem gets solved automatically. If this explanation seems harsh, look at the ‘picture’ of U’/V’ above.

Now, an hypothetical tiled tmap scanline filler would look like (sorry for the bad formatting, I still have to find a proper way to format/indent C code in WordPress :-):

void tiledtmapline (int u, int v, int du, int dv,
int run, const unsigned char * vid, const unsigned char * tmap) {
// on entry u,v,du,dv are in 8.16 format
u = (( u<<8)&0xf8000000) | ( u&0x70000) | (( u>>1)&0x7fff);
du = ((du<<8)&0xf8000000) | (du&0x70000) | ((du>>1)&0x7fff) | 0x7f88000;
v = (( v<<3)&0x07f80000) | (( v>>1)&0x7fff);
dv = ((dv<<3)&0x07f80000) | ((dv>>1)&0x7fff) | 0x78000;
vid+=run;
for (run=-run;run;run++) {
*(vid+run) = tmap [((unsigned int)(u+v)>>16)];
u =(u+du)&0xf8077fff; // addition + masking out the 1s in the gaps
v =(v+dv)&0x07f87fff; // same as above
}}

Extending to pow2 textures

Now comes the cool part. We will extend all the formulas we have developed to other texture dimensions (actually always power of 2). Let’s look at the U’ and V’ formats:

U : xxxxxXXX -> U' = xxxxx00000000XXX
V : xxxxxXXX -> V' = 00000xxxxxXXX000

bits 0-2 (LSB) of U’ and bits 3-5 of V’ are the coordinates in the single 8×8 tile. Since we always use 8×8 tiles, those fields won’t change in bitwidth. Let’s look at the remaining 5 bits of U’ (bits 11-16) and V’ (bits 6-10). 5 bits are need for 32 tiles. 32tiles*8pixels = 256 pixels.

It takes a minute to understand that by varying the number of those bits we can account for different texture sizes. With 4 bits we get 16 tiles, that is a 16*8=128 pixels width/height texture. Here are a couple of cases to make everything more clear:

• 128×128 tiled map ( = 16tiles x 16 tiles): U’ = 00xxxx0000000XX / V’ = 000000xxxxXXX000

• 64×64 tiled map ( = 8tiles x 8tiles): U’ = 0000xxx000000XXX / V’ = 0000000xxxXXX000

and so on…

So how can we handle all those cases in the formulas we wrote above? Easy: we simply need a parameter that tells us the number of bits for the ‘inter-tile’ addressing, and the corresponding mask. In formulas this will look like:

// u,v,du,dv 16.16 fixed point quantities
// bits = tile addressing bits
// mask = tile addressing bit mask
ushift = (3+bits);
umask = (mask<<(16+6+bits));
vmask = (mask<<(16+6))|0x380000;
dumask = vmask|0x8000;
u = (( u<<ushift)&umask)|( u&0x70000)|(( u>>1)&0x7fff);
du = ((du<<ushift)&umask)|(du&0x70000)|((du>>1)&0x7fff)|dumask;
v = (( v<<3)&vmask)|(( v>>1)&0x7fff);
dv = ((dv<<3)&vmask)|((dv>>1)&0x7fff)|0x78000;

and that’s all.

Here are the correct bits & mask values for the different texture sizes:

size / bits / mask
256x256 5 0x1f
128x128 4 0xf
64x64 3 0x7
32x32 2 0x3
16x16 1 0x1
8x8 0 0

The inner loop then looks like:

innerumask = umask|0x77fff;
innervmask = vmask|0x07fff;
vid+=run;
for (run=-run;run;run++) {
*(vid+run) = tmap [((unsigned int)(u+v)>>16)];
u =(u+du)&innerumask;
v =(v+dv)&innervmask;
}

And you got it! That’s a tiled texture mapper ready to handle any power of 2 texture size, subdvided in 8×8 tiles. ushift, umask, vmask, innerumask and innervmask do not need to be calculated at each scanline obviously as they depend solely on the dimensions of the texture. But a little overhead still remains; that’s true especially when you use this scanline filler in a perspective correct texture mapper that linearly interpolates every 16 pixels. One last thing to note is that wrapping is still allowed with this method.

More extensions

An obvious limit of the method I presented is that you can apply it to textures with a maximum dimension of 256×256 texels. Extending beyond this limit is not a problem: you only have to trade some bits from the fractional part, so they can be used to address more texels :)

That’s all folks!

The first post of 2007 is dedicated to the review of a book that I’ve read during the Christmas period: “Inside the Machine”, by Jon “Hannibal” Stokes, Senior CPU Editor of Ars Technica. The book subtitle says “An Illustrated Introduction to Microprocessors and Computer Architecture” and I think that it pretty much keeps its promise.

The book contains a detailed review of all the major microprocessor architectures that have seen the light since the early ’90s.

After four chapters where all the basic concepts are reviewed, from the mechanics of program execution inside CPUs, to pipelining and superscalar execution, the author jumps right into the description of one of the most widely know microprocessors from Intel, and a brand which lived for quite a long time: the Pentium.

From there on most of the chapters are dedicated to the description and comparison of the major microarchitectures and microprocessors from Intel, IBM and Motorola. In order:

• Intel P6 microarchitecture (used in Pentium Pro, Pentium II and Pentium III)
• PowerPC: 601, 603/603e, 604/604e, 750 (a.k.a G3), 7400 (a.k.a.G4)
• Intel NetBurst and the Pentium 4
• Motorola G4e (PowerPC 7450, an optimized version of the G4′s microarchitecture)
• IBM PowerPC 970 (a.k.a. G5)
• Intel Pentium M, Core Duo and Core 2 Duo

Especially interesting are the two chapters (7 and 8) dedicated to a detailed comparison of the Pentium 4 and the PowerPC G4e. Those two processors were designed with two completely different strategies, with the Pentium 4 sporting a deep and narrow pipeline, while the G4e a wide and shallow one. Jon goes into great detail on the implications of the two choices and how they impacted other design decisions and the final performance (both in raw speed and power) of the two approaches.

In between the discussion of the G5 and the PentiumM/Core, there are two interesting chapters which introduce the topics of 64bit computing (and how this was addressed by Intel and AMD) and of cache and memory hierarchy.

The chapter about PentiumM and Core/Core2 is in my opinion a bit too short: I hoped there would be more discussion on those 3 microarchitectures, especially in the light of the multi-core versions and how this impacted the architecture of the L2 cache, etc…

Jon has done a pretty good job at describing at a good level of detail some of the most important microarchitectures, always putting them into their historical context and guiding the reader in understanding how the focus of microprocessor designers has shifted from pure raw speed and high clock frequencies, to creating high-peforming low-power architectures.

Reading the book has reminded me of the cover stories I used to find in Byte Magazine which described in great details the new microprocessors as they were introduced in the market. This book helped revive in me an interest in CPU architectures and I hope Jon will consider doing a “sequel”, maybe including some of the CPUs and topis that are missing and that I think are very interesting: Stream Processors (now widely used in GPUs), VLIW processors, IBM Cell, IBM POWER6, HyperThreading and CMP (chip-multiprocessing), Sun Niagara, ARM processors, …

I wish I had such a book during my microelectronics degree courses!

Definitely a good buy.

Particularly interesting is the 2nd part of the interview with Shekhar Borkar, where some of the current techniques are detailed that are currently used to reduce power consumption and provide higher performance at a lower power cost.

[Update June 2007: I just realized the link to the interview on Intel's website are broken. The interview title was "Designing for power-The future of Moore's Law" and I can't seem to find it on another page. Will update this posting if I find it.]

While I’m at it, let me mention a few other similar links I found recently:

• Mac startup key combinations (over at Apple’s documentation site)
• Mac OS X shortcuts (at Rixstep’s blog)

1. Configuring getmail

1.1 getmailrc
Getmail is developed by Charles Cazabon and in his own words is…

“a mail retriever designed to allow you to get your mail from one or more mail accounts on various mail servers to your local machine for reading with a minimum of fuss. getmail is designed to be secure, flexible, reliable, and easy-to-use. getmail is designed to replace other mail retrievers such as fetchmail.”

Compared to the famous fetchmail developed by Eric S. Raymond, IMO getmail is easier to configure and to maintain, thanks to a clearly documented configuration file format.

In my case all I need is to:

• run getmail periodically (say every X minutes) to download emails from 2 POP3 servers (i.e. 2 different mail accounts).
• let getmail pass the emails along to procmail so they can be processed before being delivered to the final destination in my local Maildir folder.

In the first part of this series, I explained how to install getmail from the .tar.gz which is downloadable from the project home page. After that step is done, all we need to do is to prepared the configuration scripts to download emails and deliver them appropriately. I say scripts (plural) because with getmail you have to create a single script for each email server you are going to poll for new emails.

I have to poll 2 POP3 servers, which will make the configuration files very similar. For the time being I also prefer to leave email on the servers (I’m still considering this setup as a test – once I’m fully satisfied I’ll let getmail delete emails from the POP3 servers).

Here is the template of the two configuration files I have:

[retriever]
type = SimplePOP3Retriever
server = pop3.server.name
username = pop3username
password = pop3userpassword
timeout = 120

[destination]
type = MDA_external
path = /usr/bin/procmail
arguments = ("-d", "localusername")

[options]
verbose = 2
read_all = false
delete = false
delivered_to = false
message_log = ~/.getmail/logfilename

I created two of them and placed them in ~/.getmail/getmailrc-pop3-1 and ~/.getmail/getmailrc-pop3-2: they only differ for the server, username, password and message_log field.

The [retriever] section specifies the type of server (POP3 in this case), its IP address, and the necessary information to log in (username and password). I also specified a timeout of 120sec.

The [destination] section is where you specify how to deliver the mail locally. In this case I chose to use an external MDA (Mail Delivery Agent), i.e. type = MDA_external, that can be found in /usr/bin/procmail (the path for procmail in Mac OS X). Arguments to the MDA can be specified with the arguments keyword: in this case I only enforced the username (which is the local Unix user that will receive the mail).

The final section is the [options] section that allows to control the way getmail operates. Since I want to leave the mail on the POP3 server and I want to download only new messages, I added the read_all = false and delete = false settings. I also added the delivered_to = false setting but… I don’t remember why :-) (I should check back my notes!).

1.2 getmail and cron
Now that getmailrc* files are ready, we need to schedule regular runs of getmail to download email from all the POP3 servers. Even though Mac OS X has a new mechanism called launchd which offers extended functionality over cron, I still decided to setup a cron job. I will play with launchd in a later article.

First I prepared a shell script (checkmail.sh) to run getmail on the 2 getmailrc* files I previously created:

getmail --rcfile ~/.getmail/getmailrc-pop3-1 >> ~/.getmail/log-pop3-1 2>& 1
getmail --rcfile ~/.getmail/getmailrc-pop3-2 >> ~/.getmail/log-pop3-2 2>& 1

Then by running crontab -e from the Terminal I created a crontab entry to run the checkmail.sh script every 15 minutes:

0,15,30,45 * * * * <yourpath>/checkmail.sh >> ~/.getmail/log-checkmail 2>& 1

This will run the checkmail.sh script every 15 minutes and dump any error messages in the ~/.getmail/log-checkmail file. Also regularly check the log files dumped by getmail for any error accessing the servers and/or downloading emails.

Note: for the system to find the getmail executable, the directory /opt/bin should be in the PATH.

2. Configuring procmail
Why use procmail? Well first because I wanted to try it and understand it. Second because I wanted to do some processing on the incoming mails, e.g. drop them in a different folder in the Maildir directory depending on the incoming POP3 mail server.

When procmail is invoked by getmail, it will look for a file named ~/.procmailrc which contains all the necessary setup and procmail-rules to process incoming mail.

Teaching how to properly configure procmail is way beyond the scope of my articles so I suggest you take a look at the Procmail QuickStart document which is really well done and detailed.
Here I will just show you how I setup procmail to drop mails in different folders depending on the POP3 server they came from. First let’s begin with the ~/.procmailrc header, which contains some important variables:

SHELL=/bin/sh
DEFAULT=$HOME/Maildir/
MAILDIR=$DEFAULT
PMDIR=$HOME/.procmail
LOGFILE=$PMDIR/pmlog
VERBOSE=yes

The most important here are MAILDIR, which specifies the location of the Maildir folder, and DEFAULT, which gives the location of the default folder (i.e. the INBOX). LOGFILE specified the location for procmail’s logfile. I also enabled detailed messaging with VERBOSE=yes.

Now on to the rules section:

:0 c
.INBOX.backup/

:0
* ^Received:.*from pop3\.server\.one.*with POP3
{
:0
.INBOX.Account1/
}

# DEFAULT
:0
$DEFAULT

The first rule (:0 c) basically makes sure that the first thing that is done by procmail is to put a copy of the incoming mail into a backup folder (.INBOX.backup/ in Maildir naming format).

The second rule is actually a nested rule that is activated each time the mail comes from “pop3.server.one”: I apply a regular expression to the Received: part of the mail message and look for the evenlope which is added by getmail that includes the name of the POP3 server where the mail came from. This way I can do some actions (in this case a simple move to .INBOX.Account1/) on mail as they come from different POP3 servers.

Lastly if the mail doesn’t fit into any of the preceding rules, I just put it into the default incoming box.

3. That’s it!

All the pieces are now together. In part1 we have set up postfix and dovecot to act as servers to send emails to an external SMTP server (postfix) and to make the mail in the local folder available to clients through a local IMAP server (dovecot). In this article I discussed how I set up getmail and procmail to gather email from 2 POP3 servers and deliver it in the local Maildir folder (which is the folder that the IMAP server is going to access).

To test the current getmail configuration you could simply run by hand the configuration scripts, check the log files to see that both getmail and procmail worked correctly, then check your mail folder for any new mail.

I’ve not covered in detail the setup of mail clients like Mail.app and Thunderbird but since I’ve run out of time I’ll do this into a soon-to-come article.

Please let me know if you have any comments, questions or corrections to what I wrote.

Thanks!