Lesson 10. Replication

Time

This lesson takes approximately 2 hours to complete.

Goals

Use Server Admin to configure a Mac OS X server to act as an Open Directory replica Configure a Mac OS X computer to connect to a replication system Understand how LDAP data is replicated between an Open Directory master and replicas Learn how passwords stored in Password Server are synchronized between an Open Directory master and replicas Synchronize data between the master and replica servers Identify entries in the Password Server and LDAP logs that indicate problems in replicating an Open Directory master server

Because an Open Directory server provides data that is critical to the operation of your Mac OS X computers, you will need to set up more than just one server for redundancy and performance reasons. This is critical, because organizations must ensure availability in a directory-service environment. Understanding how replication works helps not only with planning and configuration, but also with troubleshooting.

In a Mac OS X Server replication system, one computer called the master is duplicated onto one or more computers called replicas, letting you move directory data closer to remote sites and provide data redundancy to eliminate service disruption. This lesson outlines the replication architecture. You will learn how to set up Open Directory replica servers and how the data—the contents of the Lightweight Directory Access Protocol (LDAP) server, Password Server, and the Kerberos key distribution center (KDC)—is synchronized between the master and replica servers.

Understanding Open Directory Replication

In a replication system, master and replica servers are dynamically and automatically synchronized to keep their contents identical. If computers go down, this ensures that services remain available and helps to avoid costly downtime. Mac OS X computers in your network can use either the master or any replica for authentication and other directory service functions. Any changes on a master or replica will be distributed to all replicas.

The process is described in the following figure:

1. The top Mac OS X computer is attempting to contact an Open Directory master whose LDAP database is not currently running.

2. Because it is configured to also look at a Replica of the Open Directory master, the Mac OS X computer can still access the LDAP information.

3. The second Mac OS X computer is simply configured to initially access the replica.

Replicas can greatly increase the robustness of your organization’s network in two ways:

• Redundancy: Mac OS X computers will automatically use a replica system for authentication when the master system is down.

• Load sharing: Replication allows Open Directory servers to be load-shared. When a Mac OS X computer seeks authentication, it can query all known Open Directory systems. The Mac OS X computer will then use the first system to respond.

Essentially, an Open Directory replica is a server that contains a duplicate of the directory data stored on a master server. If you are configuring a Mac OS X computer to retrieve records from a replica server, you should add one replica to the Authentication and Contacts search paths. If a server failure should occur, Mac OS X will use other information to reconfigure itself to connect to another replica or master server. You’ll learn more about this process later in this lesson.

Replication Topology

One Open Directory master can maintain several different replicas, and replicas can reside in the same local network as the master server or in remote networks, as illustrated in the following figure. Each replica is replicating the data from the Open Directory master with the IP address of 17.1.1.5.

When designing a replication topology, consider the following needs:

• Redundancy: Redundancy dictates that every Open Directory master support at least one replica. If the master were to go offline for any reason, the replica would take over responsibilities until the master came back online.

• Performance: In a healthy network of even up to a few hundred computers, you should not need more than two or three replicas. If a remote location requires authentication services, you should install a replica at that remote location.

• Network bandwidth: Each replica adds to the amount of traffic on a network. Although there is no software limitation to the number of replicas you can create, don’t burden the network unnecessarily.

Creating a Replica

Using the Open Directory module of Server Admin as seen in the following figure, you can define a Mac OS X server as a replica. The actual setup takes place on the computer you wish to assign as a replica, rather than on the existing master. You’ll be asked for the IP address of the master Open Directory server, the system’s root password on that server, and an LDAP directory administrator user name and password on that server. Because replication requires changes on the master server and to the master server’s LDAP database, you need both the root and administrator passwords for the master. If replication is successful, you’ll see the master server’s address entered into the Master field on the replica.

Before creating a replica, make sure that your current Open Directory master is configured as you would like it to be, including that the KDC is running and that the naming context of your LDAP database is correct. Once you have established a replication system, it is a complex process to update existing replicas to be consistent with a significantly reconfigured Open Directory master. For example, if you did not have Kerberos enabled at the time replicas were created, and then you later enabled Kerberos on the master, you would need to create a Kerberos record for each of the replicas to allow them to fully participate in the Kerberos realm. This entails creating host keys for the replicas and entering the host principal into the KDC. This process would have already been done for you if the KDC was running on the master server before the replicas were created.

Replica Creation Process

As with establishing an Open Directory master, slapconfig is the engine that drives the replica creation process. Replica creation is logged in the /Library/Logs/slapconfig.log log file on both the master and the replica. These two logs, when viewed together, completely document the automated replica creation process, which is deceptively simple in design and well thought out.

The following procedure details the replica creation process as driven by slapconfig and illustrated in the preceding figure.

1. Create a Secure Shell Protocol (SSH) connection between the replica and master computers.

Before these steps can be initiated, an SSH connection must be created. On the replica, slapconfig initiates SSH connections to the master as the master’s root user to check the supplied password for root and the LDAP administrator user on the master. If either the root or the domain administrator’s password is wrong, or the supplied administrator user does not exist in the LDAP domain on the master, the replication process cannot continue. If there are any issues with the SSH keys between these two computers, the process will also fail. The server software serial numbers on the two computers are also checked at this time. If they are the same, the replication process will fail since the serial numbers are used to identify the servers during replication.

2. Destroy any LDAP and KDC databases.

Any existing LDAP and KDC databases on the replica are destroyed and their servers are shut down.

The Password Server’s database on the replica is retained since this will contain passwords for local users on the replica computer in addition to the replicated network users.

3. Export the master computer’s LDAP database.

slapconfig stops the master computer’s LDAP server and calls slapcat to dump the entire LDAP database to an LDAP Interchange Format (LDIF) file.

4. Copy the LDIF file to the replica computer using scp, which is called by slapconfig.

5. Update the master computer’s configuration.

The master adds the replica computer’s IP address and a replication-specific user to a number of configuration files, including /etc/openldap/slapd_macosxserver.conf file, which correspond to an entry in the replica’s copy of this file. All LDAP replication between the two servers is authenticated by this user, which is not listed in the actual LDAP database.

Entries are also made in the LDAP database at cn=passwordserver,cn=config and cn=ldapreplicas,cn=config, which alert Mac OS X computers and other replicas to the existence of the replica. At this time, the master computer’s LDAP server is restarted so that Mac OS X computers can continue to use the master for directory services while the replication process continues.

6. Generate a new LDAP database using the LDIF file.

The LDIF file that was copied from the master computer is now used to generate a fresh LDAP database on the replica computer using slapadd. Once the LDAP database is created, the replica computer’s LDAP database is started.

7. Start the replicator on the master computer.

If slurpd is not already running on the master, it starts to read the change log generated by slapd. Then, using the replication user that is shared between the master computer and replica computer (created in step 5), it will add the changes into the replica computer’s LDAP database.

8. Enable Password Server and KDC on the replica computer.

The replica computer’s Password Server is made aware that it is now in a multimaster situation. At this point, Password Server will attempt to contact the master Password Server in the replication system when any password change is initiated on this particular replica.

Password Server copies the KDC database on the Open Directory master in its entirety from the master to the replica. The sso_util command is then run to generate service keys for the services hosted on the local replica. This will also change the appropriate service configuration files to allow the master and replica computers to be fully integrated into the Kerberos realm.

After the KDC is created, the replica creation process is complete. Your next task is to configure replication settings on the master computer.

Replica Configuration

Once a replica has been established, you will see the server listed on the Open Directory master in Server Admin. In the General pane of the Open Directory module, you can force a replication to ensure that a replica is up-to-date, which is useful when a replica has been offline for a period of time and has just come back online.

To customize the replication interval, services can either hold all changes and replicate once every interval or replicate whenever the directory is modified. If all your replicas are on the same local area network (LAN), you should probably set them to replicate when modified. However, if you have a number of remote networks, it is advisable to set the replication interval to a few hours to reduce traffic across wide area network (WAN) connections. Replication times set for under 60 minutes will not be recognized, and the replication will take place immediately.

Any confusion due to a longer replication interval will be mitigated by using single sign-on, since authentication is tied to a ticket instead of the user’s password. The user can authenticate to the replica with the changed password and acquire a valid ticket that will be honored by all other servers in the Kerberos realm.

Connecting the Mac OS X Computer

In order for a Mac OS X version 10.4 computer to connect to a server in a replication system, the computer must have a search path listed in the Authentication pane of Directory Access. This allows the computer to discover the replication system and build its list of replicas.

As a general rule, a Mac OS X computer should not have more than one server from the same replication system listed in the Authentication pane of Directory Access, or that computer will search the same list of users multiple times, thus negating some of the benefits of actually having the replicas in the system.

When the Mac OS X computer first contacts a server in a replication system, it downloads a list of LDAP replicas from the LDAP server to the computer and stores it in /Library/Preferences/DirectoryService/DSLDAPv3PlugInConfig.plist. In addition, the Writable key identifies the Open Directory master in this file.

The Mac OS X computer caches all known Password Servers in the replication system in /var/db/authserverreplicas.local. When the Mac OS X computer attempts to authenticate using Password Server, it looks at the Password Server public key that is stored in the user’s authentication authority and references that key in this file to find out what Password Servers can be used for authentication.

The Mac OS X computer also keeps a cache of all known KDCs in the replication system in the edu.mit.kerberos file located in /Library/Preferences. Whenever a new KDC replica is added to the replication system, that server is automatically added to the default Kerberos config file stored in the Open Directory database. Mac OS X computers will check this serialized file to see if any changes have been made, and if so, the computers will download this modified file to /Library/Preferences/edu.mit.kerberos, thus giving them access to this new replica.

Once the Mac OS X computer has the listing of all servers in the replication system, it uses the server that is specified in the LDAP configuration plug-in. If that server is unresponsive, the Mac OS X computer will progressively contact all LDAP servers, Password Servers, and KDCs. The first of each type of server to respond is cached in memory and will be used for lookups and authentication. Ideally, this will result in the Mac OS X computer using the closest server—or servers, since it is entirely possible that you will use separate servers for LDAP, Password Server, and the KDC—to it in the replication system.

The Mac OS X computer will continue to use the servers cached in memory until it restarts, its network settings change, or a cached server becomes unresponsive. At this time, it will initiate the discovery process again. If the Mac OS X computer cannot contact any of the known Password Servers in its list, it will then search over Bonjour for any other servers that share the same public key. Failing that, the Mac OS X computer will attempt to connect to a Password Server running on local computer. This last step is of benefit only when the Mac OS X computer is actually a Mac OS X Server system that has had its IP address changed.

Maintaining a Replication System

Data, including passwords, are replicated between all of the servers in a replication system. The three services that take part in replication are the LDAP server, Password Server, and the Kerberos KDC.

A process called slurpd replicates the LDAP server; Apple has not changed the standard OpenLDAP replication process. The Open Directory master hosts a single master LDAP database. Since it’s the only writable LDAP database in the replication system, all LDAP changes are pushed out by the master to the replicas.

Password Server is a multimaster system, so the version on the Open Directory master and on each replica can accept a password or a password change for a user. This change is then replicated from the originating server to the master server, and from the master server to the other replicas

Replication for the KDC is handled by the local Password Server. All changes are replicated through the normal Password Server replication process, which then updates the local KDC and reduces the potential for additional traffic on the LAN.

Because the replication architecture relies heavily on timestamps, it is good practice to keep the master and all replicas on the same Network Time Protocol (NTP) server. Mac OS X Server can act as an NTP server.

Replication Architecture

To implement a robust and complete Open Directory replication, Mac OS X Server relies on a diverse set of tools. Understanding which tools and files are used by the replication process enables the server administrator to track and troubleshoot replication problems. The following figure illustrates the processes, configuration files, and log files involved in the replication process.

Some of the more commonly referenced files that concern the replication architecture are:

• /var/run/openldap-slurp/replica/slurpd.replog: A listing of replicated LDAP database changes that is kept on the master server. All changes are timestamped.

• /var/run/openldap-slurp/replica/slurpd.status: A listing of all replica servers and the last time that they were updated for any changes to the LDAP database.

• /var/db/authserver/authserverreplicas: A file that keeps a listing of all the currently known replica Password Servers and when they were last updated.

LDAP Replication

When a change is made to the LDAP server, that change is written to a log file named /var/db/openldap/openldap-slurp/replication.log, which contains an LDIF entry and a timestamp for every change made to the LDAP database.

This log file is read by slurpd to acquire the changes pending replication. slurpd compares the timestamps of the log entries with the timestamp of the last successful replication with each replica. Replication status is kept in /var/run/openldap-slurp/replica/slurpd. status. slurpd then attempts to use standard LDAP commands to push changes out to the replicas. The master and the replicas share a user name and password, which ensures that only the master can propagate changes to the replicas.

If slurpd is unable to communicate with a replica, it will not increment the timestamp for that particular server in slurpd.status. During the next replication interval, slurpd will attempt to reconnect to the failed server and replay all changes since the last completed replication, which is useful if a server has been temporarily offline.

Only the master LDAP database can be changed. If the master server is offline, no changes to user records or other LDAP entries can be made until either the master has been brought online again, or one of the replicas has been promoted to a master.

If you attempt to connect using Workgroup Manager to a replica, you will be automatically redirected by an LDAP referral to the master server without any indication of being redirected. Similarly, if a user attempts to change the password hint or the picture for the user’s account, the Mac OS X computer will be redirected by an LDAP referral to the master server. If the master’s LDAP server is down, the Mac OS X computer will be unable to perform any changes.

Password Server Replication

Password Server engages in a multimaster replication scheme. Each server is capable of taking a change to a user’s password or password policy. Using this system, password changes can be made even if the Open Directory master is extremely busy or offline.

The Password Server that took the change will contact the Password Server on the Open Directory master. The master Password Server then replicates the changes to the Password Servers on the replica servers.

The originating Password Server refers to a list of replicas in /var/db/authserver/ authserverreplicas, and also uses this file to track when it last synchronized with a replica and whether that synchronization was successful.

When Password Server replicates, it acts in a fashion similar to that of the LDAP server; Password Server checks the timestamp of the last synchronization of each replica and then looks in its database for all changes since that time. All conflicts are settled by using the most recent change based upon the timestamp. If the Password Servers’ clocks are more than a few minutes out of sync, they will not be able to replicate. So if you are having trouble authenticating in a replication system, check the timestamp on each server.

The following figure shows how a password change is made between a replica and a master:

1. The password is passed from the replica back to the Open Directory master.

2. Password Server checks the password.

3. The Open Directory master pushes the change out to other replicas.

The security of the Password Server replication is entirely encrypted between each Password Server process. All share the same public/private key pair to both encrypt the actual database files on the local computer and also secure the communication between the servers.

KDC Replication

The following figure shows how changing a Kerberos password is propogated to each Password Server:

1. Only the KDC on the Open Directory master runs kadmind, which is the process that accepts Kerberos password and password policy changes.

All changes that are initiated from any of the Kerberos tools, such as kpasswd and the Kerberos application, will be redirected by the Kerberos realm to this server.

2. Once the changes are made on the master’s KDC, the master’s Password Server receives the changes and then attempts to replicate the changes to all other Password Servers in the replication system.

3. Once the changes are received, each updated Password Server will update its local KDC in turn.

Examples of Replication

This scenario shows the behavior of replication:

1. The Mac OS X computer starts up.

2. It attempts to use the server to which it was bound.

3. It stores the list of LDAP servers, Kerberos servers, and password servers from the replication system.

4. The LDAP server goes offline.

5. The Mac OS X computer consults its server list and attempts to connect to a new LDAP on that list.

Replication Updates

This scenario illustrates the update behavior of replication:

1. The user successfully logs in and makes a change to his or her password. The new password is stored in the Password Server database of the replica that the Mac OS X computer is currently using for authentication.

2. Password Server will now begin the replication process via the master, which pushes this change out to all other Password Servers. Because the Chicago replica is down, the password update will fail there.

3. The Chicago replica comes back online.

4. After a secure connection is established with the replicas San Francisco master, the Chicago replica checks its logs to find the last time it was updated and requests updates from the San Francisco master. The San Francisco master plays back all entries in the database that have a timestamp more recent than the Chicago replica’s last successful replication, thus synchronizing the Chicago replica with the rest of the replication system.

5. The Chicago replica comes back online.

6. The Open Directory master pushes the password change out to the Chicago replica.

Promoting a Replica to an Open Directory Master

When your Open Directory master goes permanently offline, you can promote one of the replicas to be a master. Pick a replica—ideally a server that will be fast and robust enough to handle the demands of being the master server. The promotion process does not alter the data in the LDAP, Password Server, or Kerberos databases, but it does change the services’ configuration files and causes Password Server to be rekeyed. A new public/private key pair is generated and the password database is reencrypted using the new key pair.

All of the new configuration information, including the Kerberos record and the list of available LDAP servers and Password Servers in the replication system, are uploaded into the server’s LDAP database. When Mac OS X computers connect to a server in the replication system, they will update their configurations and their lists of replication servers.

Once you have created the new master, you will need to go to each replica in the domain and manually associate them with the new master as fast as possible. Otherwise, you might run into a situation where Mac OS X computers are using computers configured with the old replication data.

Upgrading Mac OS X Server v10.3 to v10.4

In a replication system, you cannot have a mix of Mac OS X Server v10.3 and v10.4 computers. A Mac OS X Server v10.3 replica will not connect to a Mac OS X Server v10.4 master. If you upgrade your master server to Mac OS X Server v10.4, you must also upgrade the replicas.

The following figure shows four steps for upgrading a master/replica system from Mac OS X Server v10.3 to Mac OS X Server v10.4.

When upgrading, you should start with the master server; the installer will handle migrating the old databases into the new formats. After that, none of your replicas will be connected any longer. Do a clean install of Mac OS X Server v10.4 on each replica, and go through the process to configure it as a replica of your server and download the directory data to the replica.

During the migration process, the Mac OS X computers shouldn’t experience any downtime. When a server is down while it is being upgraded, the Mac OS X computer will bind to a different server.

No data is lost during the upgrade, because the master server contains all the data and copies it to the upgraded replicas. The only potential issue is if a user upgraded his or her password while the servers were being upgraded—then the Password Servers on the replicas would accept and store changes.

After the master server has been upgraded and the replicas are disconnected, password changes can’t be sent to the master Password Server. When Mac OS X Server v10.4 is installed on the replicas, the changed password will be lost, and the user’s password will revert to the old password that was on the master server.

Replica Troubleshooting

The logs are very valuable when troubleshooting the replication process. The LDAP and Password Server logs will show any attempted replications and perhaps why the attempts failed.

You can try to force replication from Server Admin; even if it doesn’t work, it might trigger errors to be written to the logs. You can also initiate replication manually from the command line with slapconfig to provide more error feedback than you get by using Server Admin.

For the initial replication to occur, you need to connect via SSH as the root user between the replica and the master server. Issues with SSH keys or other problems can prevent this from happening, so look at the SSH logs for errors.

Another common issue is time skew between the replicas or the Mac OS X computers and the authentication server. Point all of your computers to the same NTP server to ensure clock synchronization.

Directory service access control lists (ACLs) are also replicated as part of the Open Directory replication support. All nondefault ACLs will be copied to each replica each time they are updated.

Finally, as with all network services, make sure that the network is running correctly. Replication is a critical part of a robust and dependable Open Directory architecture. Apple has implemented its replication architecture using industry standard components and leveraging existing technologies. Understanding the way that replication works on Mac OS X Server is critical to planning, deploying, and supporting Open Directory in the workplace.

What You’ve Learned

• Open Directory replication applies to three primary technologies—LDAP, Password Server, and KDC—which are integrated to provide the best possible set of features while maintaining the Apple signature ease of use.

• Server Admin can promote an Open Directory replica to an Open Directory master to specify when data should be replicated to connected replicas.

• A Mac OS X computer goes through various steps during startup when it is bound to an Open Directory replica system.

• Custom directory-service ACLs are copied to each replica from the master in a replica system.

• When a user changes their password on a Mac OS X computer bound to a replica system, the password is changed on the bound replica and then passed to the master, which then updates all other replicas in the system.

References

Administration Guides

“Mac OS X Server Open Directory Administration”: http://images.apple.com/server/pdfs/Open_Directory_v10.4.pdf

“Mac OS X Server Command-Line Administration”: http://images.apple.com/server/pdfs/Command_Line_v10.4.pdf

“Mac OS X Server Upgrading and Migrating for Version 10.4 or Later”: http://images.apple.com/server/pdfs/Migration_v10.4.pdf

Apple Knowledge Base Document

The following Knowledge Base document (located at www.apple.com/support) provides further information about Kerberos authentication.

Document 107702, “Mac OS X Server 10.3 or later: Kerberos authentication may not work after changing to LDAP master or replica, or Kerberizing a particular service”

Lesson Review

1. What information does Mac OS X Server require when creating a replica?

2. How does a Mac OS X computer choose the servers it will use for authentication once it has received its lists of replicas for each of the server types (LDAP, Password Server, and KDC)?

3. When does a Mac OS X computer reevaluate which Open Directory server to use?

4. What does /var/db/openldap/openldap-slurp/replication.log contain?

5. How does KDC replication work in a replication system?

6. When will changes to a user’s record be propagated to the other servers in the replication system?

Answers

1. You must provide the IP address of the master, the root password for the master, and the directory domain administrator’s name and password.

2. The Mac OS X computer uses the server specified by the LDAP plug-in configuration. If that server fails to respond, the client will choose the first responding server for LDAP, Password Server, and KDC, while progressively querying the servers on the lists.

3. A Mac OS X computer selects the server after the computer has restarted, the network settings have changed, or the directory server becomes unresponsive.

4. It lists the LDAP data that needs to be sent out to the replica servers.

5. All changes to the principals stored in the KDC on the Open Directory master will be pushed to Password Server, which will then update the member Passwords Servers. The member Password Servers will then update their local KDC.

6. All changes to both the LDAP database and Password Server are replicated based on the replication interval set on the Open Directory master.