To understand the challenges that face us with data routing, we will familiarize ourselves with the different network segments that exist within an enterprise network for the life cycle of software. We will use the idealized version of network segmentation, as this gives us the most complete view, though rarely do all of these exist together at an enterprise. There will be a semblance of these network segments at each enterprise, though due to policies and different cultures, these segments can vary in existence, as well as name. The reason why these are important is because each segment is usually protected by a series of firewall rules. Sometimes these rules can bend, sometimes they can break, other times they are immovable objects. These rules pose challenges to getting Splunk data from the forwarders to the indexers.
For those of you unfamiliar with what a network segment is, a network segment is usually an IP address space, a VLAN, or a series of both that all machines that serve a specific purpose exist on. They are segmented from each other so data doesn't end up in the wrong place at the wrong time and cause catastrophe.
This segment is probably the most well-known, as everyone has a production environment. This is the network that all application users leverage in order to utilize a service. All of these machines are usually guarded by the strictest of processes in order to maximize user experience and uptime. Basically, this is what your customers are paying to use. This is wrapped in a high level of security in order to protect users from any unexpected impact.
In some companies (more than you'd think), testing in production is common practice. This usually leads to outages and broken dreams because the code hasn't been through any form of testing and is simply rammed into place due to leadership screaming about a deadline. This is bad practice, though not uncommon.
This network segment is where software is released into a production-like environment in order to integrate the new functionality with the old software, so as to ease the user impact. This usually has the exact same components as production, though an enterprise will invite a subset of their users (usually their premier customers) to test the functionality of the new software before it is released to production. One could think of this as a kind of beta testing environment that customers actually give feedback on in order to resolve any bugs.
This environment is usually firewalled off from any environments above and below in order to make the testing of the new software as secure as possible, and mitigate impact to only the customers who are willing to accept the risks, and provide feedback to the developers.
This is a network segment that people are paid to break the latest software as much as they can. The environment is a SIT like architecture, though there are no live customers leveraging any of this software. There are only internal staff members using the latest software releases, attempting to break everything they can in order to get the bugs out of the software.
The Quality assurance (QA) team reports their findings back to the developer, and the developer will fix all of the bugs that the QA team finds. The purpose is to clean out the software bugs before paying customers actually use the software.
This environment is usually segmented off with a very high level of security due to the amount of people truly attempting to break the software. The actions of the QA team should never reach production, as it is quite literally their job to cause outages in the software, hence the high security rules.
This segment probably has the most names and is the most confusing. Development is also known as staging, testing, dev, or coding, and is often confused with QA (because QA testers often have access here and vice versa). In most organizations I've seen, there are simply two pieces to the development environment, as follows:
- The developer's workstation
- The server(s) they put their new code on to make sure it works
Make no mistake, whatever designation your developers work in, they have almost all access to everything in it, and their personal workstations can connect and administer machines in your development environment. Just substitute development for <your company lingo> here.
The development segment is probably the most chaotic segment within a network and will almost always be completely walled off from everything else. Developers usually have free reign in this segment, so they can use all of the tools they need and connect to which resources are necessary in order to develop their software. The kick in the head is when developers come to you (the Splunk admin) and ask to put their logs in for troubleshooting, and you only have a production Splunk instance.
In an enterprise class organization, development and production don't even know each other exist, nor will they ever and there are very tight policies surrounding that rule.
This segment is a little known network segment that usually exists in every enterprise network. The DMZ is also sometimes called the App Tier and is a very small window, which is very closely guarded from QA all the way to production. There are very few systems that get authorization to be a part of this segment and this is mostly designed for monitoring.
The App Tier usually has access to all environments except the developer's workstations, and sometimes even those depending on your security and governance rules.
Due to cost efficiency, it is often cost prohibitive to purchase four appliances with all of the licensing and maintenance, and maintain each of those infrastructures separately, one in each of the environments discussed above. Using that logic, purchase a single appliance, the relevant licenses, and poke a hole in the firewall for all necessary devices. Hence what is known as the DMZ or App Tier.
These are for systems such as Nagios, System Center Operations Manager (SCOM), or Icinga, the IP address management (IPAM) system, or your inventory system, and many times, you guessed it, Splunk. Splunk can consolidate monitoring platforms such as solar winds and SCOM, and save a company, in some cases, millions of dollars in licensing. If you tell a VP that Splunk can save them six figures plus in annual licensing, and they don't have to pay anything more than what they have (depending on your Splunk license) they will give you the keys to unlock almost any door.
Sometimes you can poke as many holes as you need to for a Splunk system to work within firewalls, other times you're limited to a handful of machines. It truly depends on your company and the policies they have in place for firewall policies.