Creating a High-Level Framework

A high-level framework provides a logical structure within which you can define the major (top-level) elements of an architecture. Without a high-level framework, creating an optimal architecture design would be based more on trial-and-error than science and methodology.

Create a logical design by performing the following:

Identify High-Level Topology

Start your logical design by identifying a high-level view of the topology. FIGURE 4-1 shows a generic topology for an ISP architecture. Depending upon your customer’s requirements, the topology for your architecture may contain additional elements.

At a minimum, an ISP configuration usually consists of two key elements: point of presence (POP) and internal infrastructure.

As shown in this figure, a user connects to an ISP via a public switched telephone network (PSTN). Access between a user and POP is controlled and secured. Only infrastructure services and associated protocols/ports that are necessary to facilitate connectivities to an ISP are enabled.

The internal infrastructure represents an ISP’s internal environment where services are run and physical servers reside. Services running on the internal infrastructure are basic services, infrastructure services, and value-added services (where applicable). After subscribers connect to an ISP, they can access internal services and the Internet.

POP Topology

POP is the access point where subscribers connect to an ISP via a PSTN. The POP ensures proper authentication and authorization for subscribers accessing an ISP’s services. FIGURE 4-2 shows a typical POP topology and indicates which services are required for POP.

Services running at the POP are infrastructure services only. At a minimum, the infrastructure services necessary are domain name system (DNS), lightweight directory access protocol (LDAP), network access server (NAS), remote authentication dial-in user service (RADIUS), network time protocol (NTP), and dynamic host configuration protocol (DHCP). All of these infrastructure services are transparent to subscribers, yet absolutely necessary for providing seamless access to an ISP and its services. An ISP must have at least one POP as a point-of-access to the ISP and its services.

For a single-site configuration, a POP resides at the ISP’s data center and provides subscribers with access to an ISP when they are within the local coverage area (local POP). Additional POPs usually reside outside of an ISP’s data center at remote locations that are geographically dispersed (remote POPs). Adding more POPs enhance an ISP’s capacity for supporting a larger number of concurrent subscribers. POP structure for remote locations is very similar to local POP, with the addition of cache and one or more console servers. Because remote POP resides remotely from an ISP’s data center, a cache server is typically necessary to cache frequently accessed data, thereby reducing network traffic between the remote POP and the ISP.

A remote POP requires one or more console servers for remote servers and network devices. A console server needs to be located close by the devices managed. This requirement is based on the fact that console server cabling has short signal attenuation. System administrators at an ISP can access a console server remotely through the network.

When additional POPs are added, they are typically remote POPs to support the subscriber population beyond local subscribers and to maintain low cost and convenient access for remote and nomadic subscribers. Adding remote POPs increases access points to an ISP and capacity for supporting more concurrent subscribers. To be successful and competitive, ISPs offer roaming access in addition to local access.

Internal Infrastructure Network Topology

Within an internal infrastructure, network topology is divided into logical tiers; access through each tier is secured by separate sets of firewalls. Network segmentation simplifies an architecture design, where various modules work together to form a complex design that is highly scalable and easy to manage.

As shown in FIGURE 4-3, network infrastructure is partitioned into seven layers: de-militarized zone (DMZ) network, services network, application network, content network, staging network, backup network, and management network.

Each network layer has its own function, and each service is strategically placed on one or more layers, based on its function. Depending on a service’s function, a service runs on one or more layers, resulting in an intricate structure that facilitates optimal security and performance.

Also, network segmentation divides broadcast and collision domains. This structure isolates traffic types to enhance network response and reduce network latency. For example, a broadcast storm and resulting performance degradation are contained within a network segment so that other network segments are unaffected.

Divide network infrastructure into multiple layers to:

• Enhance network performance by segmenting collision and broadcast domains

• Enhance security through tier separation with multiple levels of secure firewalls

• Achieve N-tier architecture by functionally decomposing services

• Simplify management and troubleshooting through compartmentalizing tiers

Identify Services Within the Topology

Based on the logical network topology identified in the previous section, identify the services within each layer. (As shown in FIGURE 4-3 on page 69, network infrastructure is partioned into seven layers of networks: DMZ, services, application, content, staging, backup, and management.)

Partition or functionally decompose services into multiple layers, whenever possible, to achieve an N-tier architecture. Isolating each layer allows architectural elements to be modular, and employing this approach enhances security and manageability. Depending on the services, you can partition some into two or more layers. For example, you can functionally decompose mail service into three layers:

• Mail relay at the DMZ network to accept incoming mail from the Internet and relay outgoing mail to the Internet

• Mail proxy at the services network for users to access email service

• Mail server at the content network to contain a MailStore (back-end server for email)

DMZ Network

The DMZ network is the intermediary network between the public Internet and an ISP’s internal networks. It separates the “trusted” internal networks from the “untrusted” networks—those that are accessible to the public. FIGURE 4-4 shows a sample DMZ network.

Services running at the DMZ network are typically public services that require direct Internet access. For example, all of the services in the preceding figure require direct access because they are communicating with external services from the Internet.

For security reasons, servers from the Internet should not be able to directly connect to an ISP’s internal servers.

POP services are often integrated on the DMZ network; acting as intermediaries between the Internet and ISP, they provide open communication channels to the Internet while maintaining secured and controlled access to ISP services.

The following paragraphs describe services that typically run at the DMZ network layer.

External DNS

The external DNS is required for name resolution of external hosts from the Internet. A traditional DNS configuration works well for most ISP environments; however, a better technique is a split DNS. This technique splits DNS into internal and external domains, and it is a fine example of separating tiers by dividing the line between internal and external access.

Separating DNS into internal and external domains has several advantages:

• A split DNS prevents internal host names and IP addresses from being revealed over the Internet. This functional characteristic allows a higher level of security, safeguarding internal hosts from some external attacks, such as denial of services (DoS).

• A split DNS enhances security and preserves public IP addresses where addresses are critically diminishing.

While having a primary external DNS server reside at the DMZ is common, we strongly recommend that you move the primary external DNS server to the content network or at least somewhere on the internal networks, because in those locations it is protected by multiple firewalls. Configure only secondary external DNS servers at the DMZ network.

All zone transfers can be one-way from a primary server to a secondary server. A list of secondary servers can be specified to ensure that only authorized servers are allowed for zone transfers.

Mail Relay

Mail relay is required for relaying incoming and outgoing mail messages between the Internet and an ISP. Its primary purpose is to accept inbound email from the Internet and send outbound email to the Internet.

For inbound mail, the mail relay plays an important role in enhancing security by functioning as an intermediary layer between the Internet and the MailStore.

Hardware required for mail relay servers is lightweight and is replicable horizontally with minimal configuration change and effort. Mail relay servers can be load balanced to provide a higher level of availability.

DHCP Relay Agent

DHCP is required for dynamic network configurations for client systems. At a minimum, configurations are hostname, IP address, netmasks, domain name, DNS server(s), and default gateway. Automatically configuring these parameters is important in maintaining a centralized administration environment.

The best location for a DHCP server is at the services network. You can place a DHCP server at the DMZ network; however, for security reasons, we recommend that you include a DHCP relay agent with this configuration.

DHCP at the DMZ network is typically configured with a DHCP relay agent. This configuration can be done two ways. One way is to have a dedicated server running DHCP relay agent software. The second and preferable way is to enable a DHCP relay agent on the router to forward DHCP messages. With this configuration, a router forwards DHCP messages to connected networks without needing a dedicated DHCP relay/server on every network.

News Feeder

News feeder is necessary to receive incoming feeds from UseNet providers or upstream news servers, as well as to propagate new articles and newsgroups to downstream news servers. For security reasons, the news feeder is typically configured at the DMZ network.

News feeders are responsible for content management and storage. Hardware requirements for news storage are extremely large. We recommend that you check the following web site for the most current storage estimate:

http://newsfeed.mesh.ad.jp/flow/size.html

Although it changes rapidly, a recent estimate for news storage is that approximately 300 Gbytes of storage is required for daily news from a full feed. Due to the sheer volume, most ISPs either outsource news service to a UseNet provider or filter news feeds and moderate newsgroups to keep content manageable and minimize cost.

RADIUS

RADIUS is required for authentication of remote users connecting to an ISP. RADIUS interfaces with a wide variety of NASs and authenticates remote users against various databases, including relational databases and LDAP. RADIUS is typically configured at the DMZ network, on the same network as NASs.

NTP

NTP is required for time synchronization with external clocks. The external clock can be a hardware clock or NTP server. A dedicated hardware clock is rarely configured for small ISPs. NTP is important in ensuring that time is accurate and synchronized between servers in an infrastructure. This synchronization is critical for firewalls to maintain proper access to an ISP, based on time of the day. Also, it is necessary for the NFS server to maintain proper file access for network file systems.

FTP

FTP (file transfer protocol) is required for uploading web content from a subscriber’s system to an ISP. For security reasons, configure the FTP server at the DMZ network only.

NAS

NAS is a highly-concentrated digital modem pool with high-speed connections, such as T1 and channelized T3. Each T1 connection can provide 24 channels (23B+D), where each B channel provides 64 Kbit/sec and each D channel provides 16 Kbit/ sec. The D channel is for signal provisioning and cannot be used for connection purposes.

A channelized T3, also known as CT3, is the multiplex of 28 T1s. For small ISPs, one or more T1s are sufficient, depending upon the number of concurrent users to be supported. For large ISPs, a CT3 is more economical because a single CT3 costs less than the equivalent multiplex of T1s. Because an access server provides access to an ISP for remote users, it is commonly configured at the DMZ network. However, NAS for larger sites is usually attached to a separate access network instead of the DMZ.

Cache

A cache server can be used to cache frequently accessed data such as web content, thereby enhancing performance by reducing network traffic.

Locate a cache server close to subscribers for optimal response time. You can omit a cache server for local POP, because data resides locally at the ISP. However, for remote POP, we recommend a cache server. For every remote POP, a cache server is critical to ensure an acceptable level of performance, because data resides remotely.

Gateways

A gateway is the point of interconnect between a data network and other networks that require protocol conversion. Interfacing networks can be voice or wireless networks, as well as legacy systems. For example, a wireless application protocol (WAP) gateway is used between a wireless network and a data network, where wireless markup language (WML) is converted to/from HTML format. (A WAP gateway is usually needed for serving wireless services.) Gateways typically are configured at the DMZ network. All access to an ISP’s data network should be done at the point of interconnect to an ISP, that is, the DMZ network.

Services Network

The services network provides front-end access to ISP basic services for subscribers. Front-end servers at the services network are usually small, configured to replicate and scale horizontally with minimal changes. Front-end servers typically do not contain data and reside behind one or more load balancers, ensuring continuous services to subscribers through traffic distribution and redirection in the event of server failure.

You can install each service such as email, web, or news on separate servers; however, for management purposes, we recommend installing all services onto a single server. FIGURE 4-5 shows service components typically configured at the services network.

Configuring all services on a single server provides the following advantages:

• Single configuration for all front-end servers

• Ease of replication for horizontal scalability

• Minimum configuration and effort required for replacement

Services running at the services network are basic services and some infrastructure services that are too vulnerable to be run at the DMZ network. The following paragraphs describe each service typically placed on the services network.

Web Server

A collection of web servers, called a web farm, are front-end servers that provide access to users’ personal web pages. Web content can be either static or dynamic, depending on an ISP’s service offering.

Static web content usually resides either locally on the same server as the web server or on a centralized content server, such as an NFS server. Dynamic web content does not reside locally. Instead, it is generated by an application server; the content can reside on an application server or a content server.

In general, hardware configurations for front-end web servers are lightweight and replicable horizontally with minimal configuration change and effort.

Mail Proxy

At the front-end of the DMZ network, one or more mail proxies interface with users for accessing email. For mail retrieval, post office protocol v3 (POP3) and Internet mail access protocol v4 (IMAP4) are offered to subscribers as methods for accessing email. Simple mail transfer protocol (SMTP) is offered for sending mail.

POP3 is popular among ISPs due to the simplicity of the protocol and the modest demand on the mail server (because most of the work is done on subscribers’ systems). IMAP4 is increasingly popular among business users due to rich features and functionality, and it is becoming more common among ISPs. Similar to front-end server configuration, hardware required for mail proxies is lightweight and is replicable horizontally with minimal configuration change and effort.

News Reader

The news reader is the front-end news server where subscribers read and post news articles. The news reader often does not have local content. It interfaces with the news feeder for news articles, history, and index. Although you can install both news reader and feeder on the same server, a more optimal approach is to functionally decompose the news service into two tiers.

News readers are responsible for service requests at the front end. The hardware required for news readers is lightweight and replicable horizontally with minimal configuration change and effort.

Internal DNS

The internal DNS is for name resolution of hosts on internal networks only. The tier separation of external and internal DNS enhances security.

You can configure internal DNS servers almost anywhere on an ISP’s internal network. The most common configuration for internal DNS is placing a primary on the content network and one or more secondary servers on the services, application, or content network.

Configure internal secondary DNS servers to be forwarders. All systems on internal networks should have resolvers point to internal secondary DNS servers for name resolution. If an external name needs to be resolved, the internal DNS server forwards the query to an external DNS server.

For systems on internal networks that don’t require DNS, we recommend that they be configured to use a local hosts table. This configuration reduces the impact on DNS servers and limits security risks by only opening port 53 where required.

For reliability, strategically place multiple secondary servers on various networks to ensure service access. Couple these servers with front-end load balancers to assure availability, because DNS is critical to an ISP. If designed improperly, DNS service can cause a single point-of-failure to an entire architecture.

LDAP Replica

LDAP is a centralized method of authentication and authorization. All accesses are authenticated against LDAP, including, but not limited to, RADIUS, FTP, and email. Also, billing systems use LDAP for user provisioning, registration, and customer care.

LDAP is designed for read-intensive purposes; therefore, for optimal performance, direct all LDAP queries (read/search) to replica directory servers. We recommend using a replica for read-only purposes. Even though a master directory server can answer LDAP requests, its system resources are better used for LDAP writes, updates, and data replication.

Each replica directory server is capable of supporting millions of entries and thousands of queries per second. The directory service enables key capabilities such as single sign on (SSO) and centralized user/group management.

Most services access LDAP as read-only, such as email, FTP, and RADIUS. Very few services should access LDAP with read-write permission. Services that usually access LDAP with read-write permission are services such as calendar, webmail, billing, and directory replicas.

Similar to internal DNS servers, directory servers can be configured almost anywhere on an ISP internal network. For security reasons, always place the master directory server on the content network.

The most common configuration for directory replicas is to place them on the network where LDAP queries are intensive, such as the services and content networks. If you design multiple replicas, strategically place them on various internal networks to enhance reliability.

The hardware configuration for a directory replica is usually a multiprocessor system with a large amount of memory. For a directory master, CPU and RAM requirements can be less than directory replicas if the master is dedicated to perform only LDAP writes, updates, and data replication.

Although directory planning and analysis is beyond the scope of this book, we recommend that you approach planning for LDAP by using the following process:

For more information, refer to the following publications:

• Solaris and LDAP Naming Services: Deploying LDAP in the Enterprise

• Understanding and Deploying LDAP Directory Services

• Implementing LDAP.

DHCP

DHCP is required for dynamic network configurations for subscribers’ systems. Minimal configurations are hostname, IP address, netmasks, domain name, DNS server, and default gateway.

The dynamic configuration of these parameters is important in maintaining a centralized administration environment.

For an ISP environment, DHCP only serves dial-up users. ISP servers do not require DHCP service, and static configuration is preferred.

For redundancy, it’s a good idea to have a backup DHCP server on the internal network.

DHCP relay agents can be placed on various networks so that they relay DHCP messages to a DHCP server on the services network. For an ideal configuration, enabling DHCP relay agents at the router eliminates the need for having dedicated DHCP relay agents or servers on every network.

For security reasons, avoid placing a DHCP server on the DMZ network.

Application Network

For environments where content is dynamically generated when requested, an application server is required. It’s common to place application servers on the content network; however, we recommend that you place application servers on their own network. This configuration is essential if an ISP wants to offer application service provider (ASP) services in the near future.

For smaller ISPs, you can omit the application network if web content is static and application servers are not required. This design approach applies to smaller ISPs offering basic services with personal home web page hosting. Another option for smaller ISPs is to combine the application network with the content network.

Content Network

The content network is considered the “pot of gold” by many ISPs because it is where all ISP content resides. This network must be highly secure. Design an architecture so that no servers or services from the Internet directly communicate with services or servers on the content network. Centralize content and data on this network to increase manageability and security. FIGURE 4-6 shows a sample content network.

The following paragraphs describe services that typically run on a content network.

MailStore

The MailStore is the back-end server for email. It contains email messages for all ISP subscribers. For medium-sized and large ISPs, we recommend that some form of high availability (HA) or clustering be implemented with MailStore to enhance reliability and availability.

The MailStore interfaces with front-end mail proxies (POP3 and/or IMAP4 servers) for subscriber access to email. Unlike web servers and news servers, large-scale mail servers do not use NFS servers for storage. Large-scale mail servers have their own methods of indexing and storing email messages.

Filesystems configured for MailStore are usually configured with very large numbers of inodes in the inode table. This configuration is due to the inherent characteristic of email, that is, a large number of small files.

NFS

The NFS allows file access across a network. NFS servers are commonly used for web content and news storage.

For ISP environments with static web content, storing content locally on every web server may not be economical or manageable. In these cases, we recommend centralized storage on an NFS server.

For large ISPs, NFS servers are often configured with HA or in clusters for reliability and availability. This design principle holds true for most, if not all, servers residing in a content network.

Relational Database

A database is a collection of related information, and databases come in many varieties. Inverted list, hierarchic, and network database models are older types of database systems. These are typically inflexible and difficult to work with. In today’s world of information technology, relational databases dominate information management. Relational databases store and present information in tables and hide the complexity of data access from the user, making application development relatively simple.

For an ISP infrastructure, the relational database usually resides on the content network. This placement provides fast access to information by users and applications within the infrastructure, while maintaining a secure and centralized environment for data storage. A relational database is usually implemented in a large-scale environment where performance, availability, and manageability are critical to the business and dynamic content generation is required. For static content in a small-scale environment, an NFS server is sufficient.

Billing System

The billing system is one of the most critical infrastructure services. It facilitates new subscriber registration, customer care, user provisioning, and bill presentment.

To achieve a seamless integration, tightly integrate the billing system with the directory server. Also, relational databases are commonly needed for billing platforms and must be integrated with billing software. For small environments, the cost of the billing system can be too steep and not cost-effective to maintain in-house. Many small ISPs outsource this function to application service providers (ASPs).

Internal/External DNS

The primary DNS is required for zone transfers and zone updates. Place primary DNS servers for both internal and external domains on the content network. While having a primary external DNS server reside at the DMZ is common, we strongly recommend that you place it on the content network. Place only secondary external DNS servers on the DMZ network. (For the benefits of splitting the DNS in this manner, see “DMZ Network” on page 71.)

All zone transfers are one-way from a primary server to one or more secondary servers. A list of secondary servers can be specified to ensure that only authorized secondary servers receive zone data and dynamic updates.

When configuring DNS servers, use the following guidelines:

• Use primary DNS servers for zone updates and zone transfers only.

• Use secondary DNS servers for name resolution.

• To increase performance, point all systems to one or more secondary DNS servers for name resolution.

• For availability, use front-end load balancers for load distribution.

LDAP Master

The LDAP master directory is the core of authentication, authorization, and accounting for an ISP. Directory service is considered critical for an ISP. If not designed properly, it can be the single point-of-failure. Design the directory so that it is secure and highly available. The master directory server is usually configured with HA or clusters to ensure availability and reliability.

Replicate part or all of the directory information tree (DIT) to authorized replica servers. For optimal performance, point systems to the nearest replicas for LDAP searches and reads. They should not point to the master directory server directly for LDAP queries.

Staging Network

The staging network is for installing, developing, and testing services. Before a product, service, or upgrade is rolled out for production, it should be thoroughly tested for usability, functionality, and performance. The staging network usually consists of two main areas: an area for developing or installing software and an area for testing, as shown in FIGURE 4-7. Although they do not have to be on the same network, we place them on the same network, with different servers for each.

Developing Area

The area for developing software consists of one or more servers on which system administrators and engineers develop software for an ISP, whether it is for managing infrastructure or offering services.

The area for developing software usually does not require high-performance servers, but it must have adequate resources (CPU, memory, storage, etc.) and tools (compiler, content management, source code repository, version control, etc.). These resources are especially important when multiple administrators and engineers are modifying the same code.

Testing Area

The testing area is for simulating how an application performs under conditions representative of a production environment. Also, the testing area is necessary for use-case testing to verify that an application functions and works as expected in a scripted scenario.

We recommend that you configure the environment as closely as possible to the production environment, everything from hardware to software setup. This configuration ensures that all metrics, such as benchmarks, accurately represent and correlate with the production environment.

Management Network

The management network is a secure environment dedicated for systems, network, and security administration of an ISP. It is usually configured out-of-band, that is, the administration environment is isolated from the subscriber environment through secure access. Such segregation can be achieved through separate network switches residing behind a secure firewall. Restrict access to operations and management personnel only. FIGURE 4-8 shows a sample management network.

The following paragraphs describe services typically configured on the management network.

Console

A console server is required for managing system and network device consoles. Access to consoles must be secure; access should only be from the administration network by authorized system administrators. Locate the console server in close proximity to the servers being managed by the console, because console server cabling has a distance limitation.

Log

The log server is for managing system logs. Direct all system logs from all servers to a centralized log server. Configure the log server with sufficient storage for logs. Automate scripts to rotate and archive old logs for later analysis when required.

Grant access to the log server only from the management network by authorized system administrators. Applications that are incompatible with syslogd(1m) can be logged to local systems, but log files must be transferred via secure copy or other secure mechanisms to a centralized log server for archiving.

Boot/Install

The boot/install server is required for network booting and installation. The install server contains boot images, software images, and install scripts.

Although the boot server is different from the install server, you can configure them on the same system.

When configuring a boot server, use one of the following configurations:

• For small ISPs with few virtual local area networks (VLANs), configure a multihomed boot server (a system with more than one network interface). This approach is better than having a boot server on each and every network. A boot server is required for each network because BOOTP packets cannot be forwarded beyond the first hop.

  Caution

  Small ISPs typically configure a multihomed boot server to keep costs down; however, be advised that it is a significant security risk. Other alternatives are available. For more information, refer to JumpStart Technology.

  
  

• For large ISPs with a large number of VLANs, configure routers to relay BOOTP packets. This approach eliminates the need for having a boot server on every network segment.

Management

The management server is required for general-purpose systems and network management of an ISP. One or more management servers may be required, depending on the environment. Examples of management tasks are monitoring services, systems, and networks. Management software such as Tivoli®, PATROL® by BMC Software, Best/1®, and OpenView® may be helpful. Also, resource management software such as Solaris Bandwidth Manager (SBM), Solaris Resource Manager (SRM), and Sun Management Center (SunMC) may be helpful.

Backup Network

A backup network isolates backup traffic from other network traffic so that it doesn’t affect response time and create network latency for other services.

For most ISPs, a dedicated backup network is necessary. If an ISP has relatively little data to back up and has sufficient network bandwidth for backup traffic, a dedicated backup network is unnecessary.

Where you place backup servers depends upon the following:

• For small ISPs that do not have a backup network, place a backup server on the management network. Access to the management network needs to be open to allow back-up traffic to traverse in and out of the management network. Be aware that this configuration may have potential security risks.

• For large ISPs, place the backup server on a backup network. Only back-up traffic should be allowed to traverse in and out of the backup network. Other services should not be allowed to access the backup network directly.

A backup network does not have to be out-of-band like the management network; however, access to the backup network should be limited to backup services only, such as communication between backup agents and backup servers.

Separating backup traffic from other ISP traffic alleviates potential network contention, ensuring appropriate response times for ISP services while minimizing network latency.

Define Service Flows

To implement architectural principles and achieve an optimal design, study and identify service flows for each service. Service flow is the interaction (communication) between a client and server. Also, service flow identifies the inbound traffic type and what’s allowed and not allowed to flow into an ISP’s network.

By fully understanding and identifying service flows, you can design an architecture that is modular and that optimizes integration, security, and availability. Additionally, it’s important to understand service flow so that later you can establish firewall rules and security policies that are appropriate for the environment.

To understand communication and dependency between different services, detail the service flow for each service and document it. The following topics provide examples of service flow diagrams and descriptions of flows.

Domain Name Server (DNS)

FIGURE 4-9 shows the service flow for a split-DNS server.

An external secondary DNS server communicates with DNS servers on the Internet to provide name resolution for external host names. (External hosts should only be allowed to query an external secondary DNS server.)

An internal secondary DNS server serves only internal hosts. It provides name resolution for internal host names. When an internal DNS server needs to resolve an external Internet name, it forwards a query to the external secondary DNS server.

For security reasons, do not allow an external system on the Internet to be able to resolve internal names. Separate the environments of primary and secondary servers. Place secondary servers on networks where services reside. Place a primary server on a secure network such as the content network.

Content updates such as dynamic updates and zone transfers flow one-way. Secondary servers contain read-only data, whereas the primary server can read and write data. Requests for updates flow to the primary server. The primary server communicates zone data updates to secondary servers.

Lightweight Directory Access Protocol (LDAP)

FIGURE 4-10 shows the service flow for the LDAP server.

Applications such as email, web, news, FTP, RADIUS, and billing are designed to communicate with LDAP, which provides centralized authentication, authorization, and accounting (AAA).

All LDAP queries should be directed to replica servers, because LDAP servers are optimized for read-intensive applications. Replica directory servers are multithreaded and capable of processing thousands of queries per hour. (Actual performance depends on specific software and hardware being benchmarked.)

Replicas can answer queries and accept requests for updates; any modifications to LDAP entries flow to the master directory server. If an update request flows to a replica, the replica forwards the request to the master.

Although you could have a master directory server answer queries, it’s better for performance reasons to have the master only handle writes, updates, and data replications.

When the master updates the database, updates flow to replica servers based on how a system administrator set up the event. For example, updates may be sent to replicas on demand, on a regular schedule, or whenever an update is completed.

Dynamic Host Configuration Protocol (DHCP)

FIGURE 4-11 shows the service flow for a DHCP server.

All DHCP requests flow into the DHCP relay agent. The DHCP relay agent relays messages to the DHCP server. Requests are typically for IP address leases and some common network configurations. A subscriber’s system communicates with the DHCP relay agent server to request an IP address, and the DHCP server responds by assigning an address, usually assigned for one day (24 hours), depending upon the environment.

The DHCP relay agent server is usually configured on the DMZ network or the same network where the access server resides. The best location for a DHCP server is at the services network.

Remote Authentication Dial-In User Service (RADIUS)

FIGURE 4-12 shows the service flow for RADIUS.

When a subscriber dials in, the RADIUS server authenticates the user. The RADIUS server can have a local RADIUS database, a relational database, or LDAP for authenticating users. It can use any of these three. If you want a centralized AAA service, direct the RADIUS server to use LDAP. We recommend that you use the LDAP server for authentication because it is designed for read-intensive performance.

The access server’s configuration tells it which RADIUS server to communicate with for authenticating users.

The RADIUS server communicates with only two services: it receives requests from the NAS, and it communicates with the directory server to authenticate users.

Network Time Protocol (NTP)

FIGURE 4-13 shows the service flow for the NTP.

All systems can be configured as NTP clients. An NTP client communicates with the NTP server periodically to determine if the clock is synchronized. When necessary, it adjusts the clock. The correct time is important for the following reasons:

• When users access an ISP, the firewall must have the correct time to allow or deny access based on the current time, if required.

• Servers such as the NFS server must have the correct time to maintain file handles for mounted file systems.

• An accurate time stamp must be maintained for all access recorded in log files.

Synchronization between NTP clients and an NTP server can be done by either broadcast or multicast. In general, multicast is preferred because it has a lower overhead than broadcast.

Refer to the following web site for a list of public NTP servers:

http://www.eecis.udel.edu/∼mills/ntp/servers.html

Email Service

FIGURE 4-14 shows the service flow for email.

Users connect to a mail proxy server to send and retrieve email, using POP or IMAP.

• The mail proxy talks to a mail relay to send mail.

• The mail proxy talks to the MailStore to retrieve email for users.

For incoming email, the mail relay server relays mail to the MailStore. For outgoing mail, the mail proxy server relays email to the mail relay. For retrieving email, the mail proxy retrieves email from the MailStore for users. Users only communicate with the mail proxy for accessing email services.

Web Service

FIGURE 4-15 shows the service flow for web hosting.

To access web pages, subscribers connect to a web server via browsers. For frequently accessed data, the web server retrieves data directly from cache. Static content can be local or NFS mounted. Dynamic content is usually generated by an application server from content residing in a relational database at the back end.

For small environments with static content, NFS mounting is common. Be aware that NFS mounting can create heavy overhead for network traffic. For large environments, content is usually dynamic and, therefore, a relational database is commonly used instead of NFS mounting.

News Service

FIGURE 4-16 shows the service flow for news service.

To read or post news, subscribers connect to a news reader via browsers. The news reader communicates with the news feeder to retrieve the content. Also, the news feeder communicates with a UseNet provider for news feeds. Content can reside locally on the news feeder server or on an NFS mounted file system. Multiple news feeders can be load balanced for availability.

From a business perspective, it is usually not economical to run news service in-house. The resources (server and storage) required are expensive. We recommend that news service be outsourced to a UseNet provider.

If an ISP provides news service in-house, we recommend that the ISP moderate and filter newsgroups as much as possible. To maintain quality and performance, we recommend that the ISP remove stale news groups, restrict offensive materials (depending upon local and federal laws), and filter out duplicate news postings and expired articles.

Define Networking Components

After identifying services with the topology, define the networking components (routers, switches, load balancers, etc.) that best fit the services and overall logical design.

In the beginning of this chapter, we described an ISP network design methodology for developing an overall network topology using an N-tiered hierarchical model. In this model, the overall network structure is divided into multiple layers.

Each layer of the model has a function that is independent in purpose from other layers in the hierarchical model. However, each layer must be designed to be fully compatible and complementary to other layers. By using the hierarchical design method and keeping each layer separate, you can produce a highly flexible and scalable network. The same applies to your architectural design for networking components.

Each networking component layer provides the necessary functionality to the network. The layers do not need to be implemented as distinct physical entities. Each layer can be implemented with a combination of layer 2 (L2) switching and layer 3 (L3) routing devices. Although a layer can be omitted altogether, maintain the hierarchy to achieve optimum performance. The hierarchal network model has three layers, as shown in FIGURE 4-17.^[1]

^[1] Adapted from Designing Cisco Networks, Diane Teare, Cisco Press, 1999.

The core layer provides optimal communication between sites. The core layer is the high-speed switching backbone of the network, and this layer has the following characteristics:

• Reliability

• Redundancy

• Quick convergence

• Fault tolerance

• Low latency

The core layer primarily provides wide-area links between geographically remote sites, connecting data centers or network operation centers (NOCs).

Core links are usually point-to-point. Core services, for example, Optical Carrier-level 3 (OC-3), frame relay, asynchronous transfer mode (ATM), and so forth, are typically leased from network service providers (NSPs).

Systems rarely reside in the core layer, and we recommend not placing them there. The mission of core layer design is to focus on redundancy and reliability while bearing in mind the cost of downtime. The core layer's primary function is to provide optimal transport between remote sites. This layer is usually implemented as a high-speed wide area network (WAN), normally ATM, OC-3, or frame relay. The wide-area characteristic of the link may indicate a need for redundant paths, so that the network can withstand individual circuit outages; without redundant paths, links can create single points-of-failure. Rapid convergence of routing protocols are generally considered important core design features.

The distribution layer of the network is the demarcation point between the core and access layers. The distribution layer has many roles:

• Policy

• Security

• Media translation

• Routing between VLANs

• Broadcast and collision domain segmentation

• Demarcation between static and dynamic routing protocols

The distribution layer represents the distribution of network services to multiple VLANs within a network environment. This layer is where the backbone network is located and is typically based on fiber distributed data interface (FDDI), FastEthernet, or Gigabit Ethernet. The distribution layer is often where network policy is implemented.

Network evolution is occurring rapidly, and as newer and faster technologies emerge, existing technologies move downward in the hierarchical model. The distribution layer includes a network backbone with all its connecting routers. Typically, distribution layer devices serve a region by acting as a concentration point for many of its access tier sites. The big benefit of the hierarchal model is fast problem isolation due to network modularity.

The access layer usually consists of one or more VLANs that provide access to network services. The access layer is where almost all systems are attached to the network, usually via Ethernet, Token Ring, or FDDI. In ISP environments, including many corporate networks, the access layer is commonly where servers such as web server, email, proxy, and firewalls are located.

The principal functions of the access layer are to connect various local area networks (LANs) to the distribution layer, provide logical network segmentation, and isolate broadcast traffic between segments. The access layer is characterized by switched and shared bandwidth environment.

Routers and Switches

The decision to use L2 switching or L3 routing functionality in a network design depends on which problems you are trying to solve. These problems can be categorized as media or protocol.

Media problems occur when too many devices contend for access to a LAN segment, causing an excessive number of collisions and slow response time. Protocol problems are caused by protocols that do not scale well; they typically occur when a protocol that was designed for small networks is being used for larger networks. The general guidelines are as follows:

• If problems involve media contention, use switching.

• If problems are protocol related, use routing.

Broadcasts are used by many protocols as part of their normal operations. However, network performance can suffer from too many broadcasts. L2 switches forward broadcasts and multicasts. This approach becomes a scalability issue as flat networks become larger. If there are too many hosts on a LAN, broadcasts can cause performance degradation to the network. When broadcasts are more than approximately 20 percent of the traffic in the LAN, network performance degrades.

Routers

Routers, also known as L3 devices, are necessary for interworking communication. Routers offer the following services:

• Broadcast domain segmentation

• Hierarchal addressing

• Inter-VLAN communication

• Fast convergence

• Quality of service (QoS)

In the past, much emphasis was put on the packets-per-second (pps) forwarding rate of routers. Today, less emphasis is placed on pps because routers can process packets so quickly, especially with new switching technology provided by multilayer switches.

Like switches, routers use tables of addresses to forward packets to their proper destination. Unlike L2 switches, routers maintain tables of L3 logical addresses. Thus, router configuration is protocol-specific. Routers use specialized protocols to share information about routes and destinations among each other. With routers, broadcast packets are typically not forwarded.

Switches

Switches, also known as L2 devices, operate at the data link layer. (Refer to the OSI model for more information.) An excellent source for information is TCP/IP Illustrated, Volume 1 - The Protocols. However, in the last couple of years, LANs were revolutionized by the exploding use of switching at L2. LAN switches provide performance enhancements for new and existing data networking applications by increasing bandwidth and throughput.

The primary function of switches is to filter or forward frames. Switches work by examining frames seen on the network and by building a table that pairs the source hardware address of the frame with the switch port on which it was seen. By keeping local traffic local, switches can dramatically cut traffic on individual segments and improve overall network performance. Note that Ethernet collision packets are always filtered, but broadcast packets are always forwarded to all ports.

Ethernet is the most widely deployed LAN technology, and its use continues to grow because of its simplicity and low cost. Ethernet's main disadvantage is that it is based on Carrier Sense Multiple Access/Collision Detection (CSMA/CD), which is a bus arbitration scheme with the side effect of rapid degradation of available bandwidth in heavy traffic conditions.

A bus arbitration scheme defines the mechanism by which a system transmits data across the bus. However, this limitation can be overcome through switching. CSMA/ CD is associated with the creation of a collision domain, a concept unique to the Ethernet environment.

Although collisions are normal events in Ethernet, an excessive number of collisions further reduces available bandwidth. In reality, the actual available bandwidth of Ethernet due to collisions is reduced to a fraction (about 40 percent) of the theoretical bandwidth (10 Mbit/sec, 100 Mbit/sec, or 1000 Mbit/sec). This reduction in bandwidth can be remedied by segmenting the network using switches or routers.

Segmentation is the process of splitting a single collision domain into two or more collision domains. L2 switching can be used to segment the logical bus topology and create separate collision domains. Therefore, more bandwidth is made available to individual systems. With switching technology, attached systems receive dedicated bandwidth rather than shared bandwidth, because each system is in its own collision domain. Ethernet switches are devices that microsegment a collision domain, eliminating the impact of packet collisions.

Multilayer Switches

Traditionally, L2 switching was provided by LAN switches, and L3 networking was provided by routers. Increasingly, these two networking functions are being integrated into one common platform with multilayer switches.

Mirroring the integration of L3 networking technology into LAN switching devices, most likely WAN switching equipment, will increasingly incorporate L3 networking capabilities. As traditional L3 routers gain support for higher capacity and bandwidth, the integration of L2 technologies enables routers to achieve optimum performance levels.

New features and technology are being added to switches, for example, switches can now perform load balancing. Switching sometimes results in nonoptimal routing of packets because packets only travel on paths that are included in the Spanning Tree Protocol (STP RFC 1493), which is running to prevent broadcast storms in a switched network.

When routers are used, the routing of packets can be controlled and designed for optimal paths. Routing and redundancy in switches can be done by allowing one instance of the STP per VLAN. Using STP ensures that the network doesn’t have any loop. However, the drawback of using STP is that the convergence time can take much longer. A good network design eliminates loop where possible so that STP doesn’t need to be used. In general, incorporate switches in network design to provide high bandwidth.

Load Balancers

Load balancers are network appliances with secure, real-time, embedded operating systems that intelligently load balance IP traffic across multiple servers. Load balancers optimize the performance of your site by distributing client requests across a cluster of multiple servers, dramatically reducing the cost of providing large-scale Internet services and accelerating user access to those applications.

Load-balancing solutions, in general, are successful because they provide three fundamental benefits in server-farm environments:

Ideal for mission-critical applications, load balancers allow you to build a highly redundant and available server farm. Servers are automatically and transparently placed in and out of service, providing availability. Each load balancer itself is equipped with an optional hot-standby failover mechanism, which builds increased redundancy for the server farm system.

In today’s high-speed computing environment, such as large ISP infrastructures, load balancing with solutions such as Cisco LocalDirector, Resonate Dispatch Manager®, Resonate Dispatch Monitor®, and F5’s 3-DNS® Controller are no longer sufficient, because they represent a single point-of-failure within a large-scale infrastructure. This limitation is due to the inability of load balancers to support high bandwidth and fast switching. Load balancing is now done with multilayer switches with load balancing capabilities.

Firewalls

A firewall is defined as a single point between two or more networks through which all traffic must pass and where traffic can be controlled, secured, and logged. The earliest firewalls were routers where they segmented LANs and filtered traffic based on rules defined in access control lists (ACLs). As more businesses connected to the Internet, awareness of Internet security issues grew. The need for better security caused some vendors to develop their own solutions. Subsequently, some of these solutions were made into commercial products.

Firewalls can be network-based, host-based, or hybrid. Usually they are implemented to secure access to networks and systems that are connected to networks. While most firewalls today have many attractive features and qualities, keep in mind the fundamental tenet of security, that is, security and complexity are often directly proportional.

While some security experts debate that a firewall is just another packet filter with a pretty front-end GUI and it does not provide any substantial benefits other than packet-filtering routers, we strongly believe that security is a combination of people, technology, and processes. Admittedly, no single product or technology can provide the best optimal solution for every environment.

Nothing is absolute and there is always an uncertainty. There is uncertainty in every aspect of security. No security system is 100 percent secure. Someone, somewhere, will find a way to exploit vulnerabilities. We recommend the following guidelines:

• In addition to router ACLs and packet filters, use firewalls to control access to networks and systems.

• Use firewalls to segment and control access to network layers.

• Diversify different firewall products throughout an infrastructure. If any security exploits occur on a single firewall product, security isn’t compromised on the entire infrastructure.

• Because firewalls can be a single point-of-failure within an infrastructure if not designed properly, implement high availability at firewalls.

• For large-scale environments where heavy throughput is required, implement firewall load balancing with multilayer switches.

Intrusion Detection Systems (IDSs)

Intrusion detection systems (IDSs) can be either network-based, host-based, or hybrid. Network-based IDSs are most commonly used to examine passing network traffic for signs of intrusion. Host-based IDSs examine user and system activity on local machines for signs of intrusion. Hybrid IDSs combine both hardware and software for an end-to-end solution. Each type of IDS has strengths and weaknesses. In the following sections, we provide strengths and weaknesses of network-based and host-based IDSs.

For small ISPs serving residential users, an IDS might not have any value or return on investment. Because there are no service level agreements (SLAs) for residential users, the data are not considered sensitive or classified, and capital investment on IDS might be too expensive.

For ISPs operating on a national level, supporting both residential and business users, the demand for service availability, reliability, and security are critical to business. To proactively respond to security incidents, large ISPs should implement an IDS.

If an IDS is going to monitor front-end web servers for penetrations, then the most useful position for the sensor is on the services network, where web servers reside. If web servers are compromised, the best chance of detecting an intrusion is the target network or system.

If you want the IDS to monitor internal servers such as application servers, then the best place for the sensor is inside the firewall on the application network. The logic behind this approach is that the firewall prevents the vast majority of attacks aimed at internal servers, and that regular monitoring of firewall logs identifies attacks. The IDSs on internal networks detect some attacks that manage to get through the firewall. Some organizations like to use IDSs to monitor internal resources that contain sensitive information. In this case, the most logical place for the sensor is on the choke point between those systems and internal networks.

Note that part of the security community advocates placing sensors outside the firewall to monitor intrusions from the Internet. The firewall itself should log any attacks it stops, assuming that one enables logging and monitors the logs. What value does an IDS provide in this scenario? Intrusions stop at the firewall; therefore, an IDS outside the firewall is redundant. It probably isn’t a productive use of resources and time to install and monitor intrusions outside a firewall.

Network-Based IDSs

A network-based IDS examines passing network traffic for signs of intrusion. If you determine that a network-based IDS is the solution for your ISP customer, decide in advance where to place the sensor(s). The placement depends significantly on what kind of intrusion the ISP wants the system to detect.

A network-based IDS usually has two logical components: sensor and management station.

The sensor resides on a target network and monitors it for suspicious traffic. The sensors are usually dedicated systems that exist only to monitor the network. They have network interface in promiscuous mode, which means they receive all network traffic, not just that destined for their IP address, and they capture passing network traffic for analysis. If they detect something that looks unusual, they pass it back to the management station.

The management station resides on the management network (out-of-band), and receives alarms from sensors for analysis and response. The management station displays the alarms or performs additional analysis. Some displays are simply an interface to a network management tool, but some are custom graphical user interfaces (GUIs) designed to help system administrators analyze problems.

Strengths of network-based IDSs:

• Does not require modification of target systems. This advantage is helpful because installing additional software may exceed system capacities and degrade performance.

• Is not on a critical path for any service. IDS failure does not have a significant impact on the infrastructure.

• Tends to be more self-contained than host-based IDS. Network-based IDS runs on a dedicated system that is simple to install and configure.

Weaknesses of network-based IDSs:

• Only examines network traffic on the target network to which it is directly connected. This problem is particularly endemic in a switched Ethernet environment.

• Many sensors may be required to meet coverage on all critical network segments. Because each sensor costs money, broad coverage can become prohibitively expensive.

• Tends to use signature analysis to meet performance requirements, which means that it detects common attacks, but is inadequate for detecting more complex information threats.

• May need to communicate large volumes of data back to the management station. In many cases, monitored traffic generates a larger amount of analysis traffic.

  Many systems use aggressive data-reduction processes to reduce the amount of communicated traffic. They also push much of the decision-making process out into the sensor itself and use the management station as a status display or communications center, rather than for actual analysis. The disadvantage of this approach is that it provides very little coordination among sensors, that is, any given sensor is unaware that another has detected an attack. Such a system cannot normally detect synergistic or complex attacks.

• Is often limited on processing power (CPU) and memory. Processing engines for network-based IDSs are often not powerful enough to analyze traffic. Network-based IDSs are notorious for dropping packets if they do not have enough buffer and processing power to analyze the large amount of traffic.

• Might have a difficult time handling attacks within encrypted sessions.

Host-Based IDSs

A host-based IDS examines activity on the local server. These frequently use the system's audit and logging mechanisms as a source of information for analysis. IDS looks for unusual activity that is confined to the local host such as failed login, unauthorized file access, or system privilege alterations. The host-based IDS generally uses rule-based engines for analyzing activity.

A host-based IDS usually provides much more detailed and relevant information than a network-based IDS.

If you determine that a host-based IDS is the solution for your ISP customer, install it on a test/development system well in advance of planned installation on a production system. Even on a quiescent system, some files change regularly, so the IDS reports some changes. Some host-based systems report when a user process alters the system password file. This report would happen if an intruder added an account; however, it would also happen when an authorized user changed his or her password. System administrators need time to become familiar with the operation of IDS so that alarms can be properly diagnosed, before a production system is implemented.

Keep in mind that when using a host-based IDS, it should be monitored frequently. If the target system is compromised, an intruder can alter system logs and suppress alarms. Logging should not be logged locally to the target system. All logging should be directed to one or more dedicated log servers residing on the management network.

Strengths of host-based IDSs:

• Can be an extremely powerful tool for analyzing possible attacks. It can indicate exactly what an intruder did, which commands the intruder ran, what files were opened and/or modified, and what system calls were executed, rather than just rather vague guesses.

• Tends to have lower false positive rates than a network-based IDS. The range of commands executed on a specific target system are much more focused than the types of traffic flowing across a network. This property can reduce the complexity of host-based analysis engines.

• Can be used in environments where broad intrusion detection is not needed, or where the bandwidth is too high for sensors to handle the large volume of data to be analyzed.

• May be less risky to configure with an active response, such as terminating a service or logging off an unauthorized user. A host-based IDS is more difficult to spoof into restricting access from legitimate users and sources.

Weaknesses of host-based IDSs:

• Requires installation on the particular target system being protected. This approach can pose serious capacity and performance problems on the target system. In some cases, it can even pose security problems because the security personnel may not ordinarily have access to the target server.

• Another problem associated with host-based IDSs is that they tend to rely on the innate logging and monitoring capabilities of the target system. If the target system isn't configured to do adequate logging and monitoring, system administrators have to change the configuration, which can result in a higher level of complexity for change management.

• It is relatively expensive. Many ISPs do not have the financial resources to protect the entire network infrastructure using host-based IDS. These ISPs must very carefully choose which specific set of target systems to protect. These choices can leave wide gaps in coverage.

• Suffers to an even greater degree from isolation than a network-based IDS. It can be almost totally ignorant of the network environment. Thus, the analysis time required to evaluate damage from a potential intrusion increases linearly with the number of target systems covered.