Bob the Rebuilder

OE and Entourage, like many other e-mail programs, keep all your sent, received and filed messages in a single file, or related files. These are typically referred to as the “mail database.” The files in question are called “Database,” “Database Cache,” and “Messages,” and they can be found in the following places in Mac OS 9.x:

1. If you do not have a folder called “Documents” at the root level of your hard drive and use OE, you will find them in a folder called “Identities” inside your Outlook Express folder itself.

2. If you do have a Documents folder at the root level of your hard drive, or if you’re using Entourage, you will find a folder in the Documents folder called “Microsoft User Data.” (If you didn’t have a Documents folder before, Entourage will create it during first run.) Inside the Microsoft User Data folder you’ll find the aforementioned Identities folder for OE and an Office 2001 Identities folder for Entourage. Inside each of these folders you will find a sub-folder called “Main Identity” (unless you renamed the Main Identity to something else) and inside of that folder you’ll find the mail database files.

In Mac OS X, OE is out of the picture; the Microsoft User Data folder will be found in the Documents folder that is found inside the Home folder (the one with the “Home” icon on it that is created for each user).

The other files in these “Identities” folders deal with other functions of the e-mail software such as rules, mailing lists, schedules, and the like and are not changed in any way during a typical rebuild: only the mail database files are involved.

OE and Entourage use mail database files because they were inherited from one of the best e-mail clients of all time: Claris Emailer. That’s right—after its demise, the same people who created and refined Claris Emailer were hired by the Macintosh Business Unit of Microsoft several years ago to work the same magic for Outlook Express, then Entourage. This also explains why both products play so well with AppleScript: David Cortright, a guiding force behind all three products, is an AppleScript maven of the highest order.

Exercising Your Option (key)

Claris Emailer had a technique for compacting and rebuilding its mail database files, and OE and Entourage use the very same method. The funny thing is, few people seemed to know about this when Emailer was at its peak of popularity, and even fewer know about the feature today in OE and Entourage. So listen up, because here’s something everyone should know and hardly anyone does: to rebuild the mail database in all three programs (for those of you who still are using Emailer), hold down the Option key while launching the program.

The easy way to do this is to launch the program first, then immediately hold down the option key until you see a dialog come up (see Figure 1) that asks if you want to do a “typical” rebuild or an “advanced” rebuild. I am happy to tell you that in all the years I’ve been using these products, I’ve never had to pick the Advanced option; the only reason to do so is if the Typical rebuild fails to solve whatever problems you’re having.

Figure 1: Holding down the Option key while launching Outlook Express or Entourage reveals an important, but mostly unknown, feature.

The truth is, I rebuild my mail database every few months to prevent problems from happening. The concept is a simple one: over time, you create and send messages, read and store messages, and deal with attachments. Some you save, while others you leave with the messages they came in on. The database files inflate a bit, just like a balloon, every time you get and send e-mail.

And of course, you’re going to (hopefully) delete a lot of messages after you read them, such as spam, stupid jokes you’ve already received a dozen times, and the like. But the database just keeps expanding from all the “hot air” from the daily torrent of e-mail. And after awhile, those files get awfully big. You can tell if this is happening to you by how long it take for your e-mail program to open and then display the main browser window. What once may have taken a few seconds could now be taking 30 seconds or more. When this happens it’s time for a rebuild, which not only compacts the message database, but also your contacts as well (see Figure 2).

Figure 2: If you have a lot of e-mail to process, consider giving your e-mail program lots of RAM before starting the rebuild.

When the rebuild is finished, you’ll be told that it was successful and that it is okay to throw away the previous files which have now have the word “old” in front of them. (See Figure 3). Rebuilds can yield results that range from the subtle to the dramatic. For this article I did a rebuild and, as you can see in Figure 4, the results are pretty impressive: an almost two-thirds drop in file size. But I’ve seen even better results, such as mail database files that were 400, 500 or even 800 MB in size dropping down to under 50 MB. In any case, after the rebuild is complete the main e-mail browser window will appear, and the next time you launch your e-mail program it should appear once again with the same pep and vigor it did when you first started using it.

Figure 3: Success at last!

Figure 4: Here we see the mail database files drastically reduced in size after the rebuild. Your mileage may vary.

Trouble in E-Mail Land

If you’re getting error messages upon launch, or if your e-mail program is quitting suddenly and unexpectedly, this may also be a sign that either a typical or advanced rebuild is in order. In OS 9, you should also throw out all the Microsoft libraries so they can be replaced; in OS X, you don’t need such a complicated routine but rather can simply toss the Microsoft “plist” files. Here’s what I do when I am asked to deal with a recalcitrant Microsoft e-mail program in OS 9:

• Go to the System Folder’s “Extensions” folder and throw away any extension that starts with the word “Microsoft” or the letters “MS”
• Throw away the “MS Preference Panels” folder in the System Folder
• If a control panel is present called “Configuration Manager,” I’ll toss that one too.
• Remove either the Outlook Express preference from the System Folder’s “Preferences” folder, or the Entourage preferences found in the “Microsoft” folder inside the Preferences folder.
• Finally, I’ll restart, launch the program in question, and hold down the Option key until the rebuild dialog comes up, then do a typical rebuild.

I have found that this technique solves 95% of all OE and Entourage problems. If the problems remain, you should try the Advanced rebuild. And if that doesn’t work, consider reinstalling the software itself and also restoring the mail database files from a backup. (You do backup, don’t you?) Also, you could be having a font problem. Historically, Microsoft software programs, including Office components and Internet Explorer, are sensitive to either too many fonts, or things like duplicate fonts or font corruption. The good news is that you can fix all these things (and much more) by simply using Morrison SoftDesign’s invaluable FontDoctor 5.1.2 for both OS 9 and X.

Oh yes, one more simple but not-so-obvious thing to try if you’re having problems: in OS 9 give the program in question more RAM. Click once on the icon of the actual program itself and hit Command-I; try doubling or even tripling the preferred amount of RAM. In OS X, of course, this is no longer necessary as the OS itself figures out how much RAM things need. This is one of the reasons I like OS X!

Tons of Text

Even with rebuilding, the time may come when you have too many messages for your e-mail program to handle in an effective way. If you’ve got hundreds of folders containing thousands of messages in each one (trust me, I’ve seen it), the time has come for you to consider archiving them and getting them out of your e-mail program for good.

While it’s true that many e-mail programs use database files to keep track of everything, a real database program is far more efficient at handling this kind of thing. And the one people seem to use the most is FileMaker Pro. If you have a copy of FileMaker on your hard disk, that means you’ve also got a very cool way to archive your e-mail.

What bridges the gap between your e-mail program and FileMaker is a nifty shareware program called Entourage Email Archive by Softhing; it’s available for both OS 9 and OS X. (They also make a version for Outlook Express that is virtually identical.) According to Softhing’s site, here’s what you can do with it:

Entourage e-mail Archive (EEA) is a simple and fast utility for archiving e-mails and attachments you have received or sent using Microsoft Entourage. Entourage e-mail Archive can archive your e-mail in three different ways:

• Archive e-mail and/or attachments in the Finder
  1—Archive e-mails + attachments
  2—Only e-mails
  3—Only attachments
• Export or append e-mail in a text file
  4—Export new text file
  5—Append to text file
• Export or append e-mail in tab-text format
  6—Export new tab-text file
  7—Append to tab-text file (for this function a freeware FileMaker Pro template is enclosed in OEA folder)

Settings 1,2,3 produce a Finder-structured-folders archive where e-mails and/or attachments are grouped by day. Settings 4 and 5 produce a long ‘paper trail’ file that can be viewed with a robust text editor like BBEdit, Text-Edit Plus or Microsoft Word. Settings 6 and 7 produce a tab-text file that can be imported into computer database programs like FileMaker Pro.”

It’s those last two options that are my favorites. If you don’t own a database program such as FileMaker (remember there’s a smaller version of it inside AppleWorks), archiving each message as a text file, or in one long text file, may be enough for you. If you choose this option, you can use Sherlock to either find messages by title, or, if you have indexing turned on, by keywords or phrases inside of messages. You can even use Sherlock to find messages you might want to restore to either OE or Entourage: to do so, just drag and drop them into the main browser window of either program. Neat!

EEA features a dizzying array of options for naming and saving archived text files and the folders they’ll reside in. For more details on this, check out the concise and easy-to-understand HTML-based manual on Softhing’s site, or download a copy for yourself.

Text files are nice, but the database option is a better idea, because then you can really put the power of FileMaker to work. Previous versions of utilities such as EEA required the installation and use of obscure AppleScript add-ons, but no longer. Archiving to FileMaker now involves the exporting of selected messages or folders to a tab-delimited text file, which is then properly imported by the EEA-supplied FileMaker Pro template, which requires FileMaker Pro 5 or higher.

Using EEA is easy and enjoyable. You first launch either OE or Entourage depending on which you’re using; for best results, make sure either program has enough RAM assigned to it under OS 9. When first launching either EEA or OEA on top of your e-mail program, you’ll be greeted with the opening screen (see Figure 5.) From here on out, we’ll stick with EEA and Entourage, but rest assured the same also applies for OE and OEA.

Figure 5: Make sure to launch OE or Entourage first before running the EEA e-mail archiving application.

If this is the first time you’re using EEA, you’ll want to go to the File menu and click on Preferences. Decide first what kind of archiving you’d like to do, then click from tab to tab and make the appropriate selections (see Figure 6). Here, I’ve told EEA I want a new, tab-delimited export that will be imported into FileMaker. I could also say I want the new data appended to a pre-existing text file. EEA conveniently lets me pick the creator of the resulting text file; in this case, I’ve picked Tex-Edit, my favorite shareware text editor. This is important because, in OS 9, SimpleText has a 32K file size limit. Bets are, I’m going to exceed that even with one folder full of imported messages.

After setting Preferences and hitting OK, you’ll be returned to the main screen. For most archives, simply click on the Include Subfolders checkbox, then click on the icon on the left of the pop-up menu to have EEA gather all the folder information from Entourage. You’ll then see all of your Entourage folder names displayed; select the one that has messages you want to export (see Figure 7). You may only pick one folder at a time to export, but if you would rather export every message in every folder, click on the All Structure checkbox instead.

Figure 6: Before starting your first archive session in EEA, make sure to visit the Preferences menu and tell EEA exactly how you want your archive done.

Figure 7: Like magic, here’s our selection of Entourage folders now appearing inside of EEA.

Clicking on the “Destination” icon below the Entourage icon lets you pick where you want your text file saved, and how you want it named. If you’ve decided on a straight Finder-based individual message archive, this is where you’ll decide where the files and folders should be saved. Finally, hit the “Process” button to get underway (see Figure 8).

Figure 8: Once underway, EEA shows you its progress as it exports every message in a given folder to a master text file.

After the export is complete, it’s time to open the FileMaker Pro template provided by EEA and import the data you just exported from Entourage. The first time you import a tab-delimited text file into EEA, you’ll be asked to match up the fields from the Export file with the fields inside the EEA template. They should already match up (see Figure 9), but if they don’t you can simply drag the fields on the right side (the template side) so they match up with the fields on the left side (the text file side). This also gives you some latitude in what you decide to import: if you want header information, go ahead and click on “Headers” so that the arrow appears.

Figure 9: The last step in the Archiving process involves importing the tab-delimited text file you just created into FileMaker. Make sure the fields on the right match the fields on the left; you’ll only need to do this once.

Figure 10: Here’s what the EEA FileMaker template looks like. The information in every field is editable.

If you don’t care about headers, leave them turned off. It’s up to you exactly how much information on each message you want imported. When you’re happy with the field names, click on “Import,” and quick as a wink, your e-mails will now reside within FileMaker. Here, you can very quickly search and sort to your heart’s content; copy, paste, or export messages as you see fit; and otherwise play with the data (see Figure 10 for a sample page).

When you’re satisfied that the e-mails in question have been imported properly, you may return to Entourage and delete the original e-mail messages from the folder in question and then go on to other folders, or quit both Entourage, EEA, and FileMaker. Keep in mind that e-mail attachments are not archived to FileMaker…only the reference to them is. If you delete a message out of Entourage that has an attachment, that attachment will also be deleted. Therefore, if you decide you need the attachment, you’d better save it to disk first; you could, if you wanted to, create a folder and call it “Archived Attachments.” Then when you see a reference to one inside the EEA FileMaker template, you’ll know where to find it. Of course, not every attachment is worth keeping, so perhaps having the reference to it is enough for you. Also keep in mind that the EEA FileMaker database file is completely editable, so you can add or remove information from selected entries at will.

Now It’s Your Turn

Compacting your e-mail databases will make your e-mail program snappy and responsive again, while archiving will give you greater power over your old e-mail, as well as peace of mind. That’s important, because should the worst happen and your entire e-mail database is somehow destroyed or corrupted and there is no backup, at least you’ll have whatever you’ve saved in a virtually indestructible FileMaker file.

Now it’s time to hit the gym, and to put your e-mail program on a diet!