The Entourage Help Blog

What is the Database?

That’s Database with a capital D.

The Database is the single file that Entourage uses to store your mail, attachments, calendar, contacts, tasks, notes, projects, categories, links and other data that you see in Entourage. If E-mail is your life and you use Entourage then the Database file is the single most important file on your computer.

You can find the Database in its default location ~/Documents/Microsoft User Data/Office 200x Identities/Main Identity/ where “ ~ ” is your home folder. If you don’t see the Main Identity folder then you’ve probably renamed it to something more familiar to you. Inside the Main Identity folder is a the Database file.

This file can be huge for some folks. I’ve seen reports of up to 25GB in size but often it is at least 500MB or more for the typical Entourage user. Select your Database file and choose the Get Info command from the File menu to see the size of your database.

Why is my E-mail and everything else stored in one file?

The simple answer is that it’s a database (lowercase d). Databases are optimized for quick access and fast searching.

Think of a database like a book. A book can have a table of contents so that you can quickly skip to the part of the book you want to read. A book can also have an index so that you can look up specific words that my not have a topic in the table of contents.

Organizing data on your computer is done the same way. The Finder provides you a table of contents in the form of folders such as the Applications folder and the Users folder as well as files such as the Read Me files that come with most applications. It also offers an index in the form of Spotlight so that you can find specific data anywhere on your computer by searching for a key word.

Microsoft created the Database back when hard drives were increasing in size, the amount of data that people were accumulating was growing exponentially and Spotlight didn’t yet exist. It was a solution to a problem.

So what’s wrong with still using the Database?

It doesn’t scale well.

Huh?

That means it doesn’t necessarily work well at all sizes. This is one of my favorite examples:

Consider a the sizes of a flea, an elephant and a planet.

A flea can jump 600 times its own height. But if you enlarge it to the size of an elephant then it will crush under its own weight.

An elephant is much sturdier and can carry its weight on four massive legs. But if you enlarge it to the size of a planet then it will collapse into a sphere.

A planet can orbit a star but when reduced to the size of a flea it can create a black hole and absorb the star.

Whereas the Entourage Database was once like a flea, it’s now growing to elephantine sizes. It’s not scaling well and this is a risk to your Entourage data.

What can go wrong?

While the Entourage Database is a robust database, it can get corrupt. Microsoft includes the Microsoft Database Utility to verify and repair problems but it’s not perfect and never will be. It can not fix problems caused by the Mac OS, hardware or other applications.

Even if the Microsoft Database Utility could fix all the problems of the Database, the time to fix it is directly proportional to the size of the Database. That means the larger your database, the longer to fix the problems. And for many folks this could mean hours. And if you’re running low on hard disk space then you may not even be able to repair the Database at all.

The only way to ensure your data is safe is to maintain backups—copies of your data before it was corrupted. Backups are an afterthought to many computer users because they are themselves time-consuming, require additional software, additional hardware and require management. The advantage of backups is never realized until they are actually needed, which is why many folks live on the edge and never back up regularly.

Time Machine is a huge boon to Mac OS X users because it makes backups simple but it can not backup the Entourage Database properly. Time Machine and other backup applications must copy each file to a different location. To do this, the file must be closed. However, the Entourage Database is always open because of the Microsoft Database Daemon.

A daemon is a process on your computer that runs in the background. The Microsoft Database Daemon is constantly using the Entourage Database so that it can remind you of events, create Spotlight caches for searching and monitor the health of the Database itself. Even if Entourage is not running, the daemon is still using it.

If Time Machine or any other backup utility were to copy the Database file while it’s in use then the backup would be corrupted (although your original is still just fine).

What else is possible?

This topic was discussed in the TidBITS Talk mailing list in early January and a lot of pros and cons were raised about whether or not to use a database.

One list member noted that Entourage 2008 contains an important new AppleScript addition to export archives from Entourage. Exporting to a smaller file and then deleting items from the database would be more manageable. However, the Entourage dictionary does not include a Compact command to reduce the size of the Database file. It also requires some knowledge of how to either use AppleScript or implement someone else’s script.

Another idea discussed was to completely eliminate the Database and store all messages as single files like Apple’s Mail application. Mail stores its messages in .mbox files in ~/Library/Application Support/Mail/account name/.

These .mbox files are really folders that can be opened by simply deleting the file extension and double-clicking the folder. Individual messages are stored in a Messages sub-folder. These can be backed up or retrieved one message at a time instead of having to deal with an entire database. While this is easy to implement and improves the speed and reliability of backups, searching can get slower when iterating through more individual files. Additionally, the meta data about your messages—the categories and links—are not preserved unless you store that information in a different file.

Maybe Entourage could store its data in multiple databases, each with its own index. This would reduce the risk of losing all data if the Database gets corrupt, decrease backup/restore times and decrease the amount of time to repair mail stores. While this looks promising, it still means a proprietary set of database files that only Entourage could use.

What’s the best solution?

Ditch the Database!

Again, that’s Database with a capital D.

I’m not a software engineer but the most logical solution I’ve seen so far is to ditch the Database and instead create a hybrid database/file solution. The data (the actual mail messages, contacts, calendar events, etc.) can be stored as individual items with the meta data (categories, links, etc.) stored in a smaller database file. The database can keep track of the location of the data without actually storing the data within itself.

Files can be maintained individually.

• One bad sector on a disk would not be allowed to compromise more than one file (one mail message).
• A full 2GB database file would not need a backup just to preserve a new 4k mail message or the deletion of a mail message or any item.
• The time to incrementally backup and restore would be proportional to the amount of data that has actually changed.

Meta data can be preserved in an smaller, external database.

• Categories, tags, links and other data that I myself associate with my data can be preserved.
• If any meta data is corrupted then my data (mail messages) are not adversely affected.

Security of my data must be top priority.

• The less that changes then the less opportunity for corruption.
• Less data that has to be copied for backups means faster backups and that means my data is secured in minutes and not potentially hours.

Hard drives are getting bigger and people are storing more data. Simple backups are still the most economical means of protecting that data for most users. Microsoft must change its method of storing your Entourage data to better scale with the increasing amount of data you need to maintain. If they don’t change how they store your data then you run the risk of losing all of it because backups are either too unwieldy or you can not back up your data in a timely manner.

Do you find backups for your Entourage Database complicated or do you avoid them? Let Microsoft know your concerns about the security of your data by sending them feedback through their website http://www.microsoft.com/mac/suggestions.mspx?product=entourage.

For additional information about Time Machine and using it to back up your Entourage Database read Diane’s post about Entourage and Time Machine as well as her post for an Alternative method to use Entourage and Time Machine.