Exchange 2000 introduced the concept of the storage group (SG): a set of databases (each one of which can contain mailbox or public folder data) that share a common set of transaction logs. The impetus behind this design is simple: you can mount or dismount individual databases, which means you can back up, restore, repair, defragment, or otherwise work with one database without affecting users whose mailboxes are in other databases. Exchange 2000 and Exchange Server 2003 Enterprise Edition servers can have up to four SGs, whether they’re standalone or clustered, with up to five databases in each SG. The Standard edition of these products allows you to create one MAPI public folder database and one mailbox database (although you can create additional non-MAPI public folder stores, and even multiple SGs, if you like). Each of these databases is limited to 16 GB in size; Enterprise Edition databases don’t have set size limits.

Exchange Server 2003 offers a special additional SG known as a recovery storage group (RSG); the RSG can be used to mount any database from any Exchange 2000 or later server in the organization. Once the database is mounted in the RSG, you can copy mailbox data out of it and into another database, making it easy to set up “life support” messaging during a disaster recovery.

Exchange stores much of its message data in files known as EDB files, after their file extension. These files contain all of the message, folder, and attachment properties that Exchange exposes via MAPI and WebDAV; apart from the messages and mailboxes themselves, there are a large number of indices and tables that are dynamically created and updated to provide improved performance for various types of queries. Exchange has always used a custom database engine known as the Extensible Storage Engine, or ESE. ESE provides a low-level interface that the Information Store service builds on; ESE is mostly concerned with the content and arrangement of the 4 KB pages that make up the individual database files. ESE is optimized for storing MAPI messages, which have an interesting quirk: the set of properties stored for two different messages may be very different, making it hard to efficiently construct a relational database structure to contain them.

Among its other duties, ESE is responsible for implementing transaction logging (more on that in the next section), providing an interface for reading and writing individual database pages, and managing the process of performing backups using the ESE backup API or the Windows Server 2003 Volume Shadow Copy Service (VSS).

One of the biggest scalability limits in Exchange 5.5 came from Exchange using its own native storage format (the Transport Neutral Encapsulation Format, or TNEF) to store items in the store. Unfortunately, TNEF bears approximately zero resemblance to the Multipurpose Internet Mail Extensions (MIME) standard, the way mail messages are actually transported around on the Internet. That means that any time a client used SMTP to submit a message to the store, it had to be converted from MIME to TNEF. Worse yet, any time an Internet-protocol client wanted to use POP3, IMAP4, or HTTP to get a message out it had to be converted back. Worst of all, the content conversion always took place, even if it wasn’t necessary! This ate up so much CPU and I/O bandwidth that Microsoft decided to take an alternate approach for Exchange 2000.

The streaming media, or STM, file is that approach. It provides a way for the Exchange components to stream data in on-the-wire formats like MIME, without breaking it up into 4 KB chunks and storing it in an EDB file. That means that a 10 MB PowerPoint presentation or a 25 MB QuickTime movie will be stored with no content conversion and without being split up. Content will no longer be converted unless it’s absolutely necessary; when a message is received through an Internet protocol, the data is passed through to the STM file, and the native MIME streams are left intact. If another Internet protocol client attempts to read the data, it is streamed directly out of the native content store without ever being converted. If a MAPI client such as Outlook 2000 attempts to read the data that resides in the native content store, the store process converts the MIME stream into a set of MAPI properties on demand and passes the data to the Outlook client. Better yet, this on-demand conversion takes place entirely in memory unless the data are changed as part of the process (for example, if an Outlook user opens that PowerPoint presentation, edits it, and saves it again).

The store uses a very aggressive online defragmentation process to keep as much contiguous space as possible free in the STM file. Interestingly, STM files are like magnetic poles: they don’t exist by themselves. STM files are always paired with EDB files, and the two files share a set of transaction logs, are backed up together, and generally combine to act as a unit. The two are even added together to determine the store size, when calculating the 16 GB database size limit in Standard editions of Exchange.

The fundamental idea behind logging is simple: the logs store copies of all transactions. These stored transactions can be played back later to restore a corrupted database, or even to retry a transaction that didn’t complete successfully. Logging is a mainstay of relational database engines because it provides a backup mechanism for transactions to help preserve the ACID properties of the database:

Although Exchange treats the logs for a SG as a single entity, it’s actually a set of files, each of which is exactly 5 MB (that’s 5,242,880 bytes, for you purists) in size, even if there aren’t any transactions in it. If you see a log file that’s any other size, it’s probably corrupted. The IS maintains a log file named edb.log. When the log file fills up, the IS service renames it, using a sequential hexadecimal ID (the first file is edb00001.log, the second is edb00002.log, and so on). These renamed log files are called generations ; edb.log represents the highest, or most recent, generation. Note that just because a log file is full, doesn’t mean its transactions have been committed—all commitments happen according to a process I’ll describe in a moment.

The log files contain a number of tidbits that are useful if the logs have to be played back during server recovery, including the full path to the database files, information about which generation of log data is in the file, a signature and timestamp for the data, and a signature for the database. This header information enables the store to make sure that each log file is replayed into the correct database file, and to balk if you do something like try to restore files from one machine onto another. Of course, the log files also contain information about the transactions themselves. For each transaction, the log records the type of transaction (i.e., whether the transaction represents a change, a rollback of a previous change, or a commit of a previous change). These transactions record the low-level modifications to individual pages and tables within the database.

You want to create a new SG on an Exchange server.

The Enterprise version of Exchange 2000 and Exchange Server 2003 Servers supports up to four SGs, each containing five databases. The Standard edition allows zero or one public databases and zero or one mailbox databases; normally these are in the same SG. Exchange Server 2003 also allows you to create and use a recovery storage group, but the presence of an RSG doesn’t count against the four-SG limit.

Even though you can easily create multiple SGs, you probably shouldn’t do so until you have used all the database slots in the first SG on your server. Every time you add a SG, you also add an additional instance of the ESE code, which adds a significant amount of memory overhead. The one time to violate this rule is when you need to create a separate set of transaction logs; since logs belong to the SG and not the database, if you want multiple separate log sets, you’ll have to create multiple SGs.

When you do create a new SG, you’ll note that there are settings for enabling circular logging and page zeroing. I’ve already explained what circular logging is and when you should use it, but page zeroing is a different matter. When a database page is deleted from an ESE file, the page header is marked with a tombstone that indicates that the page is free, and it goes into the whitespace pool for reassignment later. Its contents remain intact, though, which means that confidential data could potentially be recovered by anyone who can get access to the database file or backup copies. In most environments, this is a low risk; however, if you turn on page zeroing, each deleted page is filled with zeroes when it’s marked as deleted. When planning for performance, keep in mind that page zeroing is a highly disk-intensive process and has a substantial performance impact.

The VBScript code uses CDOEXM to actually create the new SG. The CDOEXM StorageGroup object requires that you specify a URL at which the new SG will be created; although you can specify the complete URL, it’s easier to specify the new SG’s name and the server where you want it created, then retrieve the first named SG on the server and use its URL to construct the URL of the new object. By default, Exchange will create the SG log files in the default log path; you can move them later using the method described in Recipe 6.9. As with most other CDOEXM objects, you have to use the SaveTo method to finalize your changes.

Recipe 6.9 for moving storage group logs to another disk location, Recipe 11.11 for using recovery storage groups, MS KB 319218 (HOW TO: Add New Mailbox Stores in Exchange 2000), MS KB 821748 (HOW TO: Add New Mailbox Stores in Exchange Server 2003), MSDN: CDOEXM documentation, and MS KB 824126 (How to use Recovery Storage Groups in Exchange Server 2003)

You want to delete a storage group on an Exchange server.

Unlike mailbox or public databases, the SG object itself doesn’t really have any useful data in it; it’s a container that points to a set of transaction logs and databases. Accordingly, when you remove it, the corresponding directory object is removed but nothing changes on the target server’s filesystem. You can easily recreate a deleted SG, since the databases and log files are left intact (but orphaned) when you delete the SG itself.

You must first delete (or move) any mailbox or public folder stores that are housed in the SG. If you attempt to delete a SG that still contains active databases, ESM will complain; if you’re doing it from within a script, the script will fail.

Recipe 6.1 for creating storage groups, Recipe 5.6 for moving mailboxes, Recipe Recipe 6.6 for deleting databases, and MSDN CDOEXM documentation

You want a report that shows you what storage groups exist on your server, and where their database and log files are located.

SGs have a variety of properties available through CDOEXM, including the names (both common and distinguished) of databases in the SG and the log and database file paths for the databases. You can view and set these properties by using code like that shown in the solution; however, you won’t be able to change the log or database file locations by changing the associated properties. Instead, you’ll have to use the MoveLogFiles() and MoveSystemFiles() methods (see Recipe 6.9).

Recipes Recipe 6.1 and Recipe 6.2 for creating and deleting storage groups and Recipe 6.9 for moving databases and log files; MSDN: IStorageGroup class documentation

You wish to create one or more additional mailbox databases in an existing storage group.

When you create a new mailbox database, Exchange generates a new, empty EDB file and fills it with the structures necessary to contain mailbox data. It also generates a matching STM file, which is empty until you start moving or creating mailboxes in the new store. By default, newly created stores aren’t mounted; ESM will ask you if you want to mount the store, while you can use the Mount method on the mailbox store interface to mount a newly created store programmatically.

The previous VBScript example uses the CDOEXM MailboxStoreDB interface to create a new database in the first SG on the server. You can easily modify the code to create databases in a named SG by specifying the URL of the SG instead of looking up the first one. If you do not wish to mount the store, simply comment out the last two lines of the VBScript example.

Recipe 6.5 for creating a public store, Recipe 6.6 for deleting a database, Recipes Recipe 6.7 and Recipe 6.8 for mounting or dismounting a database, and MSDN: IMailboxStoreDB

You want to create an additional public folder database to host your applications.

Exchange 2000 and 2003 support multiple public folder hierarchies (also known as top-level hierarchies (TLHs) or public folder trees). When you initially install Exchange 2000 or Exchange Server 2003 Server’s, you automatically get a TLH that’s accessible to MAPI clients; this is where user-accessible public folders are stored. There’s a catch, though: there can be only one TLH accessible to MAPI clients. Clients that use NNTP or the WebDAV extensions to HTTP can use non-MAPI TLHs, but you must first create at least one non-MAPI TLH before you can add more than one public folder store to a SG. If you don’t have a public store at all, when you create one, it’s automatically associated with the default MAPI TLH.

Recipe 9.5 for forcing replication of a public folder hierarchy, Recipe 9.6 for getting properties of a public folder tree, Recipe 9.7 for creating and removing public folders, and MS KB 255077 (HOW TO: Create a New Public Folder Tree and Store in Exchange)

You need to remove a database you no longer need.

When you delete a database file, you’re severing its relationship with its parent SG, as well as deleting the corresponding AD object. However, neither ESM nor CDOEXM removes the actual database files from the filesystem. This is a useful protective measure, since you can easily create a new database with the same name, dismount it, and replace its files with the original database files.

Recipes Recipe 6.4 and Recipe 6.5 for creating mailbox and public folder databases

You need to mount a database so users can connect to it.

You must mount a database before users can connect to it, or before you attempt to back it up using the ESE backup APIs discussed in Chapter 11. When you mount a database, ESE performs a quick integrity check on the database header; after the database is mounted, if there are pending transaction logs, they will be played back. This can affect the speed of mounting the database (although, interestingly, dismounting a database always takes roughly the same amount of time, no matter how large it is). If Store.exe is able to mount the database, it will post event 9523 to the application event log.

Recipe 6.8 for dismounting databases

You need to dismount a database so you can perform maintenance or integrity checks on it.

Dismounting a database is simple, but it’s required before you copy the database files using anything other than the Exchange backup APIs. That’s because the dismount process sets a flag in the database header that indicates that the database was cleanly dismounted; if that flag is not there, Exchange will refuse to mount the database again until the header has been repaired with eseutil’s /r mode.

Recipe 6.7 for mounting databases, and Microsoft Technet article “Top 10 Database Mounting Issues and Their Solutions” (http://www.microsoft.com/technet/prodtechnol/exchange/2003/hd.mspx)

You want to relocate the files for a database, or the transaction logs for a storage group, from one disk volume to another.

For best performance and recoverability, you should keep your Exchange transaction logs on a different physical disk or array than the Exchange database files. This gives the ability to have up-to-date logs to play back in the event of a database drive failure, allowing administrators to restore to the point of failure. The Exchange 5.5 Performance Wizard would attempt to figure out which disks to put its database (singular) and log files on, but Exchange 2000 and later dump everything in program filesexchsrvrmdbdata. You can, and should, relocate your databases and log files to enhance performance and recoverability. However, before you can move a database, it has to be dismounted; and before you can move the log files for a storage group, you’ll have to dismount all of its databases.

The scripts for this recipe use the CDOEXM interfaces for the MailboxStoreDB and StorageGroup objects to move databases and transaction logs. In both cases, the first SG’s URL is used as the basis for constructing the URL to the target SG or database. The database moving script dismounts the database if it’s mounted when the script is run, and mounts it after the move, because you cannot move a mounted database.

MS KB 821915 (How to Move Exchange Databases and Logs in Exchange Server 2003), MS KB 257184 (XADM: How to Move Exchange Databases and Logs in Exchange 2000 Server), and Exchange Server 2003 Performance and Scalability Guide (available from the Microsoft web site at http://www.microsoft.com/technet/prodtechnol/exchange/2003/library/perfscalguide.mspx)

You wish to determine how much available “whitespace” exists within a given database.

Many administrators are keen to understand how much whitespace is contained in their Exchange databases. This space includes database pages that were once allocated but have been freed. In most cases, an EDB/STM file pair will grow steadily over time, but if you’ve done something to shrink the actual amount of mail data in the file pair (such as moving mailboxes to another server or purging old mail), there may be a substantial amount of disk space allocated to the file but unused by Exchange.

The amount of whitespace is especially important to know in Exchange Server Standard Edition when the database size is approaching the 16 GB limit. Unfortunately, there is no built-in utility to determine the amount of whitespace, so the simplest way to find this information is simply to look in the Application Event Viewer logs for event 1221, an informational event that is logged each night after scheduled online maintenance. Here’s a sample from one of our servers:

Event Type:    Information
Event Source:    MSExchangeIS Public Store
Event Category:    General 
Event ID:    1221
Date:        1/4/2005
Time:        3:15:00 AM
User:        N/A
Computer:BATMAN
Description:
The database "First Storage GroupPublic Folder Store (BATMAN)" 
has 1 megabytes of free space after online defragmentation has terminated.

If you run the command-line recipe, you will notice that it’s not always the same number as that reported with event 1221; eseutil.exe provides a more accurate estimate (at the cost of having to dismount your database).

Recipe 6.16 for defragmenting the database, and MS KB 186291 (XADM: Cannot Determine Free Space Available in Exchange Database)

You need to determine the anchor log file (i.e., the last consistent log file) to replay logs.

If you have an offline backup that you wish to recover, and you want to replay transaction logs to restore up to the point of failure, you need to identify the low anchor log file. This is the first in the series of log files that must be replayed to restore all available transactions back into the Exchange database. You must ensure that you have all log files from the low anchor log file forward to the log file that contains the point-of-failure transactions (the high anchor log file). To know which log files you need to restore from your offline backup, you’ll need to be able to find the anchor log file; once you’ve done that, you can move on to restoring the offline backup, as described in Recipe 11.12.

Recipe 6.8 for dismounting the store, and MS KB 296788 (Offline Backup and Restoration Procedures for Exchange)

One of your databases has become corrupt and you want to restore it using only the transaction log files.

If you have an unbroken chain of transaction log files and PRIV1.EDB becomes corrupt, you can quickly and accurately recover the data store by dismounting the database and removing (or renaming, which is much safer) the existing CHK, EDB and STM files. When you attempt to remount the database, the IS will create new, empty EDB and STM files, then it will begin sequentially playing back log files to restore the contents of these files.

Note that the anchor log isn’t relevant here; in order for Exchange to restore the contents of the database accurately, you’ll need all of the log files. There’s no way to tell Exchange “start with log file #5 and go forward from there”; in the case of an empty EDB and STM, the anchor log file will always be the first one. Because this approach requires you to have every log file created since the database was created, it’s pretty difficult to find circumstances where it makes sense. In most cases (and assuming a reasonable level of backup planning), it will be easier to fix the database by reloading it from a backup and playing back the needed transaction logs, as described in Recipe 11.4.

Recipe 11.4 for restoring a database, MS KB 278958 (How to Use Log Files to Rebuild the Priv1.edb File), and MS KB 296843 (How to recover an Exchange 2000 Server database after error -1216)

You want to find out which mailboxes are in a specific mailbox database.

Finding the mailboxes in a given database is simple, provided you know the DN of the mailbox database itself (which, in turn, is easy to get from the SG object, which is easy to get from the server object) By querying on the homeMDB attribute, you can select the subset of mailboxes in the database of interest. The script for this recipe takes this process up one level by iterating through all of the SGs on the server, then looping through each mailbox database in each SG and displaying the mailboxes that database contains. One caveat, though: a mailbox can exist in Active Directory, but until the user logs onto the mailbox or the mailbox is sent mail, it will not be created in the store.

Recipe 6.3 for listing the SGs on a server, and Recipes in Chapter 5 for working directly with mailboxes

You wish to enable circular logging for your Exchange Server transaction logs.

Since a new log file is created whenever the current one fills up, log files can potentially take up a large amount of space on your disk. One solution is to put them on a dedicated disk set; another is to enable circular logging. Normally, every log file is kept until its transactions have been committed; the files are usually purged when online backups are made. When you enable circular logging, Exchange will keep a fixed number of log files, rolling from one to another as transactions arrive. The default number is four, but Exchange may use extra log files if a large set of transactions arrives. As the fourth log file fills up, Exchange will commit transactions from the first file; when the fourth file is completely full, all transactions will be flushed from the first file and it will be reused. However, since these additional log files are never deleted, a very busy server can still use more than the default 25 MB circular logging allocation.

Since circular logging overwrites transaction logs, it can be useful as a temporary space-saving measure on a drive or array that is running out of space. However, the cost is that since a full set of transaction logs don’t exist, up-to-the-minute log playback is not available in the case of media failure. I don’t recommend using circular logging on mailbox servers; however, on front-end, public folder replica, or SMTP servers, it’s often a sensible way to cap the amount of disk space used by those servers (recall that the SMTP service requires a mounted mailbox store to send NDRs for failed messages).

Toggling circular logging is easy, since it’s controlled by a single attribute on the SG. All the script does is set this attribute, then use the Save method to persist the change, which takes place immediately.

MS KB 147524 (How Circular Logging Affects the Use of Transaction Logs) and MS KB 258470 (XADM: How to Modify the Circular Logging Setting)

You want to change the schedule for Exchange Server’s scheduled maintenance processes.

As with most other databases, the ESE holds on to space once it’s allocated and reuses it internally instead of returning it to the filesystem. For example, if you create 50 10 MB Word documents and delete 20 of them, you’ll regain 200 MB (20 * 10) of space. If your Exchange stores grow to 24 GB and you delete 1.5 GB worth of mailbox data, the stores will still take up 24 GB on disk, but Exchange will be able to recycle the 1.5 GB of free space within the database. The Exchange databases are like a balloon with a one-way valve attached—you can blow it up bigger and bigger, but it doesn’t normally get any smaller unless you do something purposeful to let some air out. Online maintenance performs several tasks to optimize the ways in which this whitespace is reused. These tasks include:

Recipe 6.16 for adjusting the online maintenance schedule as part of shrinking the database, and MS KB 159306 (XADM: IS Maintenance Tasks Not on IS Schedule), MS KB 159197 (XADM: Controlling Folder Index Aging)

You want to reclaim whitespace from your Exchange database, so you need to take it offline and defragment it.

In normal operation, it is almost never necessary to defragment the database as it will simply grow again as more messages are sent and received. Some activities can return large amounts of space to the database whitespace pool. For example, moving large numbers of mailboxes off a server will mean that the space formerly used by those mailboxes is available; if you move enough mailboxes, this savings can be significant. To run an offline defragmentation, Microsoft recommends that you have 110% of the database size available as free disk space—for a 40 GB database, then, you should have at least 44 GB free.

When you use eseutil’s /d option, it systematically scans through the EDB and STM files for the specific database, rearranging pages to make all of the tables, indices, folders, attachments, streamed objects, and messages contiguous. Normally, it does this by following a simple process:

Step 3 can take a significant amount of time, so you can instruct eseutil to skip it with the /p option. This forces eseutil not to copy the temporary file or overwrite the existing file; you’ll have to do that manually before you can mount the database. Alternatively, you can use the techniques in Recipe 6.9 to change the database paths to point to the newly defragmented file.

Recipe 6.9 for changing database and log file paths, Recipe 6.17 for shrinking an oversized Standard Edition database, Recipe 11.2 and Recipe 11.3 for completing a full online backup, MS KB 328804 (How to Defragment Exchange Databases), MS KB 192185 (How to defragment with the Eseutil utility [Eseutil.exe]), and MS KB 254132 (Eseutil /d defragments the database and the streaming file)

You have the Standard Edition of Exchange 2000 or Exchange Server 2003, and your mailbox database has reached the 16 GB limit so it keeps dismounting.

The standard version of Exchange has long had a 16 GB limit; the only real change between Exchange 5.5 and Exchange 2000 and Exchange Server 2003 is that the older version allowed you to have only one database, and the newer versions allow one SG with two databases: one private and one public, each limited to 16 GB. The simplest way to override the capacity limitations is to install the Enterprise version of Exchange over the existing version; this is a simple and nondestructive operation that follows the same process described in Recipe 2.12. Failing that, you can use the procedure above to temporarily increase the size limit to 17 GB, which buys you some time during which your users should clear out some of their old mail. Step 9 will force the store not to keep any previously deleted items in the deleted item retention pool, but they won’t actually be removed until the next time the online maintenance task runs. Once it runs, the store will purge all previously deleted items, and you’ll have back as much free space as you can get without defragmenting the database.

Recipe 2.14 for upgrading to the Enterprise edition of the same version of Exchange; Recipe 6.7 for remounting the mailbox database, Recipe Recipe 6.10 for determining when online maintenance has finished, Recipe 6.15 for controlling the online maintenance schedule, and MS KB 828070 (Exchange Server Mailbox Store Does Not Mount When the Mailbox Store Database Reaches the 16-GB Limit).