A Lost Enterprise Client

As a product manager for a leading vulnerability management solution, I thought I had seen every type of client until a recent trip at the end of 2010. A potential new client, a Fortune 100 company, was using a point and shot vulnerability assessment solution for the past six years. They needed to grow into an enterprise solution that covered all aspects of the business and embarked on an RFI to gather criteria. This is where it all began; the security department wrote the RFI as a one-page document without a budget, management approval, and without the knowledge of legal and procurement. That was our first tip something was wrong. The RFI arrived with no cover letter and not even formatted on company letterhead. There was no document control number or even a disclaimer. My first instinct was to delete the email and move on. I should have followed my business instinct. But, at the assurances of my enterprise account manager, I answered the questions and began the pilot.

With any enterprise pilot, having hardware and solution prerequisites in place is critical for success. If the client cannot provide a basic lab environment with minimum hardware and software, you are doomed from the moment you put your foot in the door. A lesson every systems engineer learns the hard way at least once. Well, the Friday before we went onsite, the client still had no hardware for our pilot but assured us everything would be great Tuesday morning when we arrived. Flights are booked, the weather was holding, and we took the trip.

As a backup plan to loading the solution, I learned another lesson a few years ago. Have a plan B. In this case, I had a virtual machine with all of our solutions loaded, with demo licenses keys, and an evaluation version of the operating system and database. If the client did not have the hardware and software I needed, surely, I can find a workstation with enough RAM and CPU to run VMware Player and my plan B.

So, we arrived onsite that Tuesday morning and of course, security has no knowledge of our visit. In addition, the salesperson does not even have the correct building, so once we get in, we had to fumble around the campus to find the right entrance. Did I mention it was 20°F outside?

In either case, we get there, meet our contact, and are directed to the lab. The newest server they have is from 2001 in the lab. It is running tons of other software, takes forever to boot, and is not a candidate for an enterprise pilot with only one hard disk and no available resources. The box is maxed out.

So, we hunt for another desktop and implement plan B. We find a desktop from about 2003 with 6GB of RAM and Windows XP x64-bit. Surely this should work, but no. Without the CPU supporting VT technology, VMPlayer cannot run a VM an x64-bit OS with 8GB of RAM. Strike Two. We identify a newer desktop with much less RAM, but it meets the other requirements. Three hours later, we are uncompressing an 80GB VM on an underpowered VM.

Now, I pull the salesperson aside and tell him the bad news. He, of course, does not care and tells me to make it work. I was a systems engineer before I was the product manager, then vice president, and CTO; and I knew better even back then. But what should I do? Punt?

Well, midway through the day, I got the VM running and did some initial scans. Everything ran very slowly and made the product look awful. I am not sure what is worse: having a buggy solution, or a solution that shows poorly because of the operational environment.

Now, I must add one piece regarding personnel throughout this process. During the ordeal of finding a system, we got passed through three different individuals that acted as our escorts. No one would take ownership of our presence. Is that another red flag or what?

Anyway, we finished doing some scans, ran some reports, and left for the day. I thought for good. We never did a demo for management. We never wrapped up the pilot and never asked what was important. There were no sales questions and no follow-up plan. Doomed from the start, and I was a sucker for listening to the account manager.

So, it is now the end of the year. Sales wants every dollar in. The account manager informs me that they are really serious now and need to go back onsite to close the deal. My “Spidey” sense kicked in again and said this is nuts. Yup, nuts. But, as a good PM, I listened and went.

We arrived onsite with snow and foul weather. We were escorted to a conference room and spent about two hours with one individual demoing the solution. Then they left. We were left alone for lunch and almost three more hours before they return. We wrapped up the lesson in less than an hour. We never met the executives in charge of the project, and they wouldn’t give us any answers on procurement. This was our second visit, and the account manager was certain we could close this deal. He was wrong. We left, and I battled bad weather flying to my next city for a speaking engagement. This was doomed from day one. The client went dark, and we never found out what they purchased, if anything.

Lessons learned:

• Make sure any pilot for a company this size is funded and approved.

• If paperwork is incomplete, find out why.

• Make sure all prerequisites for a pilot are met before arrival.

• Always have a Plan B, especially for travel. This is kind of like the rules from Zombie Land.

• Make sure you are speaking to the right individuals in the company, especially procurement and management.

• Trust your instinct and never let a salesperson talk you into a sure win.

Just a Win

Once in a while, you win a client that will do anything for you. It is almost like having a best friend for life. If you solve a problem for them, one that makes a difference, nothing will ever jade them. In about 2004, I assisted a client with the rollout of our endpoint solution. It was a raw product back then. Barely out of the 1.0 release. The management console was honestly very immature and the deployment tricky as heck. But, we had a paying client, and the boss wanted this thing rolled out and operational.

The one thing about this client, compared to so many others, is that they would bend over backwards to get something done and working. Hardware, no issues. The right security profiles on workstations, “No problem.”

So early one morning, we began deploying agents and had to stop almost immediately. All of the agents being deployed were creating a message storm of events. No idea why. As I mentioned, the management console was a wet noodle, and all we could do is watch logs scroll by. It was rather ugly.

So, after troubleshooting some to settle the noise down, we determined that all of the deployed agents were spamming messages about three different IP addresses on the network. What was odd was that the message was the same in every event.

Our liaison isolated the IP addresses to medical equipment in a corresponding building. Our next step was a short visit to see what they were and what they were doing. After arriving we noticed that all three pieces were identical GE pieces of medical diagnostic equipment and that they had Windows XP with no service packs, firewall, or anti-virus running on them.

All three of these systems were infected by a worm. They were scanning the network for other devices to infect and when they attempted to infect a machine with our endpoint solution on it, it generated an alert. That is what we were seeing.

Critical pieces of medical equipment in a health care-providing facility were infected and trying to infect other devices on the network. Can you imagine the CISO's face when he saw this?

We promptly disconnected each piece of medical equipment from the network, configured them to run in stand-alone mode (put the results on a floppy versus communicating them to a PACS system on the network), and observed the message storm completely drop to zero.

This is one find and in one place were the technology worked so well, saved the client an embarrassing problem, and made the decision makers and stakeholders of our solution look like kings. We won a client for life, made a huge difference in their network, and potentially saved someone’s life. It does not get much better than that.

Lessons Learned:

• It only takes one event to win or lose a client. A big win will keep them for life.

• No matter how wet the code is, do not dismiss errors and problems quickly. It may just be working correctly.

• When your solution solves a real problem, make sure everyone knows. If you do not, others cannot learn from your win.

Just Too Much to Manage

I hear all the buzz about application whitelisting, and application blacklisting. Philosophies on how to best protect applications and control what executes. Now before companies began building these massive MD5 hash libraries to support these initiatives, clients have to register applications one at a time with their host based firewalls and application contol solutions. When a new version came out, and when a new version was patched, the IT department had to register the version for it to work and let it communicate on the wire.

So, in visiting a client in NYC only a few years ago, they told us that across the several thousand assets they had, they had three versions of Microsoft office in their environment. Some versions were professional, some were basic, and some were business based on the time of acquisition and user needs. All of these versions had various service packs and security patches loaded. At the end of the meeting, we estimated over 100 different versions of files for MS Office that would need to be managed by the host-based firewall since the client had no patch management or standardization in process.

Now, add to that all the third-party applications, and custom applications they developed (and there were a few), they needed to manage a few thousand MD5 hashes for all the desktops, and the list kept growing. Admittedly, they knew this was nuts and wanted a better solution. We discussed the technology my company had to offer, but they wanted to still do everything with hash control. They just wanted a better way to manage them versus their current tools and could not accept a better, newer technology that did not require management at this granular level.

Needless to say, they stayed with the current technology they had and added more people to manage the security of these devices.

Lesson Learned:

• If you cannot convey a better technology that the client will accept, the ROI does not matter.

• The client must be willing to change their opinion on what they are using. Otherwise, they will just buy a bigger version of what they already have.

• Throwing people at a problem rarely works. It works best for labor-intensive applications.

• Change is good, but the client has to be willing to change. Sometimes they are just looking to justify continuing what they currently do.

Obsolete

An overseas visit was requested by an enterprise client to review odd vulnerability data and a worldwide architectural deployment. Upon arriving at the client, I expected things to be in pretty good shape, but we did have a few vulnerabilities that required global exceptions. After cleaning up the data, we began looking at the highest-risk assets and low-hanging fruit for remediation. This led to some startling findings.

The first was an obsolete system running Windows NT 4.0. This server was critical for business operations and client fulfillment. It was also deemed end of life by Microsoft several years earlier. After determining who owned the system, why it was never updated, and even decommissioned, it was identified that the hardware was provided by a third-party contract and the owner had no obligation to update it or provide any security. If the device was compromised, it could bring operations to a dead halt. The only mitigation was to install inline security devices and configure heavy ACLs around access and configuration to the device. The server could not be updated or replaced, and the contract provided no provisions for any disputes over the equipment outside of an SLA to repair. There where no security provisions.

The second finding involved domain controllers. The scanner reported Anonymous Share Access on several mount points on their domain controllers. The share name was a little nonstandard, so we decided to investigate. After connecting from a lab workstation that was not on the domain, we saw the user directories for every account in Active Directory. There was no security on any folder, and we could browse any folder, including the CFO. So, what was a good security professional to do? We copied a couple of financial documents with sensitive information and sent it to the CSO. That caused a ripple effect that I never heard the full extent of, but I did find out the share was placed there in lieu of a back agent in order to back up all user data. The lack of security was so that the remote connection could read all the directories. In doing so, it left everything wide open for any user to browse, copy, read, and even delete files if they wanted to. The share was promptly removed and another process put in place instead.

Lessons Learned:

• Vulnerability exceptions are acceptable in any business as long as proper mitigation and risk acceptance procedures are in place.

• When negotiating contracts with third-party vendors for services and equipment, make sure security and maintenance are a part of the contract.

• Verify that other processes and services do not jeopardize security.

• Something like backup procedures needs to be just as secure as any other process. Remember the data in a backup could represent all of the company and not just one finite element.

Complex Is Best

Have you ever seen a complex architecture that just looks like overkill? Sometimes it is needed and sometimes absolutely required based on all the use cases. One enterprise client had one of the most nightmarish environments I have ever seen. Multiple sites, low, unreliable bandwidth, and frequent problems at each site with workstations and servers were just some of the problems I found. To top it off, the datacenter at each location is limited to one rack and adding servers and appliances is not an option. So, we were left deploying a software solution that shared resources with another server.

So back in headquarters, we set up a standard management server and connected all of the scanners. After a few days of test scanning, we found the results streaming back to the server over poor satellite links was choking the bandwidth and causing business interruptions to other applications. So, our first step was to schedule the scan job off hours. The second step was to schedule the data to only upload in a very small window in the middle of the night. Then scans could run, not cause runtime issues due to a restricted infrastructure, and the data upload when other operations are dormant. After all was said and done, this complex architecture had multiple scan windows scattered throughout a month and rolling data upload windows scheduled back to back for all locations. A whiteboard carried the initial calendar and schedule that was implemented, and a complex architecture was reigned in and made the difference. The client was happy, the architecture worked, and the environment benefited from aggressive patch verification and PCI compliance.

Lessons Learned:

• Do not be scared of complex architectures. Sometimes they are necessary to meet the business requirements. Just make sure they are necessary and not overkill.

• Flexibility. Using tools that are flexible for a deployment, whether it be software or appliances, or has settings to manage data flow and jobs at a granular level is important to meet unique business requirements.

• Planning. Project Management is key to a complex rollout. Planning, scheduling, and documentation can make the difference and make sure all responsible parties are on the same page.

Forfeit the Game

In watching high school sports with my kids, I have seen a few games that were forfeited because the other team did not show or had too few players. This only happened a few times, but in the business world, it has happened too many times. One large successful deal my team won was because we were the only player willing to play. The requirements were straightforward, and the mission was to show up onsite and ready to install the pilot. All of my competitors sent software and instructions, or appliances, and offered webinars to get things started. None offered personnel to come onsite. And, why? I honestly did not blame them, but in order to play, they needed to show up and participate. This was a customer requirement and a statement they made that would help determine the winner. So, why did no one show up? Because the pilot was just below the Arctic border in the middle of December. That is where the customer was, and that was where I needed to be. So, four connections later and a flight that lands only every other day, I made it to my destination.

As with any trip, the first step on landing in absolutely freezing weather is to bundle up and dress warmly. That worked. The next step was to find my rental car. Well, the same person that helped me off the plane rented me a car. This was a really small town. In fact, the waitress at the hotel restaurant told me men outnumber women four out of five to one in the town, and she could not wait to leave: for anywhere but there. Anyways, I got to my rental, and it had a cracked windshield. In fact, all the windshields on the rental cars were cracked. I went back to my deboarding agent and rental car staff member and queried why? By this time, I was already freezing. The agent informed me that all the roads are dirt and rock and that they kick up stones and break the windshields frequently. They are left this way, so should I still care? The cars have traction regardless of ice and snow build-up. I knew this was going to be a fun trip. In order to play, you need to show up.

So, I proceeded to my hotel and parked. I went to check in and was asked if I plugged in my car. Now, I live in the sunshine state of Florida and have only heard of plugging in my car if it was an electric or hybrid. So of course, I said “What?” The polite attendant told me that you need to plug in the engine heater overnight to keep the engine oil warm or the car will not start in the morning. No wonder my competitors did not show up. So, I brazed the cold again and plugged in the car into the outlet on the outside of the building.

It was now time for dinner and sleep.

The next day I arrived at the client, did the installation, and proceeded with a very normal pilot. After all was said and done, we won the business and I froze every moment I was up there. I flew back the next day, 16 hours in transit, and had a purchase order on the salesperson’s desk within the next few weeks. The client did not even try the other solutions. They wanted personal attention in the Arctic, and after the client had lived there so many years, he had the authority to bring someone there to help make his decision. That made all the difference in the world.

Lessons Learned:

• If a client wants an onsite, saying no may lose the business. You need to know if you push back, will it hurt you.

• If you are going to engage a client, a half-assed effort will fail. You either give it your all or not at all. You will not win business on a whim, or half energized approach.

• Regardless of the place or weather make the trip. Just because it is too cold or too hot, it should not stop your trip.

• Being personal and meeting your client face to face builds relationships. Sometimes this is absolutely required to close the sale.

Listening Skills

You may have a client for years. They may have your product and solutions fully deployed and in everyday business. but if they want to get rid of you, they will. As a fact, every vendor can be replaced, and no vendor is so entrenched that they cannot be removed. So, let’s start with listening skills. A client environment was partially deployed with the solution and was hitting technology problems with regard to scalability and reliability. An executive meeting between the vice president of product management and vice president of enterprise sales was scheduled to review the issues one at a time with executives and key team members. A one-hour meeting turned into almost four hours, and every item was documented and commitments made to as many of them as possible. We listened to their problems, commitment to resolve the problems, and presented a plan to fix everything possible. Now they never told us we would be thrown out, but we knew that would happen.

Now over the course of the next six months, we had several maintenance releases, and we called the client each time and told them what we fixed, and what we added to solve his problems. Fixing his problems and delivering them was about 90% of the answer. The other 10% was showing him what we did and confirming it was the right fix. This gave us a deeper trust in the client and listening paid off.

Lessons Learned:

• Addressing your client’s issues is important. You need to step back, listen, and document what they need and figure out how to deliver.

• Listening without delivering will only set you up as one of those vendors that do get removed.

• Showing your client the progress you made is just as important. It shows you listened, cared, and were willing to work with them.

• Never underestimate a client’s ability to remove a vendor from operations.

Contractors

Shortly after 9-11, my presence was requested at a secure facility in South Florida. With security being the most paramount concern in everyone's mind, I hightailed it down the Florida Turnpike to my destination. Needless to say, the police officer that pulled me over for doing 90 did not share my enthusiasm for security. After a nice hefty fine, I arrived onsite and began my work. I was delegated to work with a contractor that was completely unfamiliar with my tools and had an open source product he preferred better. He loaded it and showed me all the capabilities he liked and wanted. Needless to say, he was trying to convince me what he wanted was better than what the client had paid for and wanted installed. After his brief demo, we began installation and usage of the solution the client wanted. Now this facility was so secure, I was not allowed to touch the keyboard. The contractor needed to do all the typing, and simple things like passwords just did not seem to work. He blamed my tool and again reinforced why his was better. Did I say this was a secure government facility? In either case, after troubleshooting for more than a day, I finally had him re-enter credentials, and things started to work. We had barely enough time to finish the setup before we did a demo for the officers in charge of the facility. The demo went fine, and we were fully installed, operational, and the contractor was trained.

Before I left, I had a debriefing with a senior official at the facility. We discussed the delays, and I mentioned the open source software the contractor loaded. He was completely lost in the conversation and stated that this was a secure facility, and no one should ever load unapproved software on the network. He promptly called the contractor in and confronted him with the accusation. He attempted to do the installation and pleaded his case. I was asked to leave and not to worry about anything else.

I gathered my things, turned in my bathroom hall pass, and saw the contractor leaving too. I said goodbye and found out through a rather awkward conversation that he was terminated for installing unauthorized software. I have never heard from either of them again.

Lessons Learned:

• A contractor never has the final say and must follow the rules of the company. They may have other restrictions as well that go above and beyond direct employees.

• Change control, highly secure environments, and proper procedures can get anyone fired if they are intentionally violated.

• If something is not working as expected, check your passwords. Fat fingers can cause many of the problems.

• If you intentionally sabotage a project, it will bite back. This is a simple lesson: what goes around, comes around.

• Nothing said behind closed doors is truly ever behind closed doors.

The Rogue Device

Before Hurricane Katrina hit the Mississippi area, there were a plethora of casino barges on the Gulf Coast and river. One of these clients had a datacenter located nearby and a major corporate office. After a fairly extensive pilot, and evaluation of casino devices, servers, and workstations, we proved a scan would not disrupt operations or devices like slot machines and cash changes that were connected to the network of the casino floor. After a routine scan one day, we started noticing what appeared to be rogue IP addresses. They were in the normal IP range but had no reverse DNS look-up, and the operating system was reported as a Turtle Beach MP3 Player. That in and of itself was very odd. After some rather complex tracking of the IP address, since the entire casino floor was a flat subnet, we found network-based MP3 players plugged into the casino network by two employees sharing large quantities of music and using the backbone to copy gigabytes at a time.

The devices were promptly confiscated and the owners identified. This was a quick identification of rogue devices and illegal activity, but we truly got lucky in finding them and emphaized the power of a good discovery.

Lessons Learned:

• Having rogue devices can represent a risk to the business that is unacceptable.

• Users having physical access to the network need to be controlled and locked down when appropriate.

• Illegal contents on the network, like MP3 files, can make the company liable for the contents.

• Identifying rogue devices is critical for sensitive networks.

• Using cameras and other security devices in critical wire closets will booster your security profile and prevent tampering.

The Big Fish

Some salespeople go after the biggest fish. The biggest enterprise client, the biggest partner, or the biggest OEM deal. Winning that business, getting a fat commission check, make you feel that the effort was worth it, but to a small business, it can be incredibly destructive. The needs of the biggest client can easily consume the resources of a small business and ultimately make the entire deal a losing proposition. Consider winning a deal with the big three server and desktop vendors for a piece of software that they will all embed. The cost to localize the product, provide technical support and training, and service the client can easily bury a business. The cost of business for the biggest fish must consider adding resources, the pains of growth, and what it will take to service them without making them your only client.

Now, consider you are the small fish and went after the biggest fish in the world. Well, we did. And we won it. Seven years later we have learned a ton. Here are some of biggest lessons learned:

• Stick to the contract. If you give too many free services away, it may be your undoing to keep the client happy.

• Define roles and responsibilities clearly, so there are no missed steps.

• Any service-level agreements must be achievable. Do not agree to them just to win the deal.

• Providing services that have never been done by you before must be estimated and gauged for viability. The worst scenario is the service is completely unobtainable in the time allotted.

• As my boss always states, crawl, walk, run. Do not expect to manage the biggest client from day one running.

Rootkits Anyone?

My enterprise account manager received a panic call from a prospect on a Friday afternoon requesting our presence first thing the following week. Based on that call, we determined the client was in real hot water due to a recent malware infestation. The scope of which we did not know. So, we booked flights and made arrangements to be there promptly and early on Monday morning. The early bird gets the worm, right?

As our meeting started, we learned the domain controllers throughout the organization had been infected with a rootkit. This was shortly after the big news storm about Sony installing rootkits on PCs using audio CDs. Well, the client did the right thing and did not bother to clean these machines. They started the painful process of creating a new domain and reinstalling all of the domain controllers one at a time and migrating users over. A process that took months to complete.

So, why the sudden urgency to have us onsite? Well, with the news about Sony, the executive team decided they wanted some sort of protection capabilities, so this type of exploit would never happen again. Can you image reinstalling everything from scratch? Drivers, software, users, policies, settings, and even restoring user data after it has been scanned? This was well before widespread DevOps and virtualization. The cost and time were enormous, and they did not want to repeat the mistake since they never determined how the machines were infected in the first place.

My account manager and I presented a textbook case for our solution, and I can proudly say that even after eight years of loyalty, they have never had a problem even close to this again.

Lessons Learned:

• A prospect in need can be an easy win. Just be honest and solve their problems with real results. They will stay loyal and reference over and over again.

• When you have an infection with the magnitude of rootkits on domain controllers, there is no safe or guaranteed way to remove the malware and be certain you are in the clear. A complete reload is the only proper course of action.

• If you follow the lead of your prospect, including their urgency, it shows your ability to empathize with their issues and provide solutions. If you dismiss their urgency and claim they are just crying wolf, they will not trust that you understand their problems.

• No one ever wants to wait for a fix to a problem. People want the pain to go away immediately.

Not the Only One

At one time to create leads, my company offered our endpoint protection product for free to end users. One of those downloads was from a CFO for a very unique vertical that had sensitive data from many, many clients. This is way before GDPR. He was running the product on his corporate machine when Conficker broke out. We received one of those desperate calls for help, and we immediately reacted with an onsite visit and a solution to meet their needs. During our fact-finding mission, we discovered every server in the environment was compromised, since nothing was being patched, and that the majority of desktops had also been compromised except for the CFO's machine. He suspected a problem when the agent repeatedly showed Conficker Alerts from critical servers in his environment. Our solution was protecting him from exploitation, but for the rest of the systems, it was too late.

So, we received a purchase order for the entire environment in very short order and began a rollout of key assets as the IT department began patching and disinfecting all of the systems, one at a time.

Lessons Learned:

• Endpoint protection is only truly effective if everyone is covered. Missing any systems leaves them open to threats.

• If your solutions are sending critical alerts, you need to investigate them. Dismissing them as false positives or noise will make you complacent to real threats.

• If the CFO never used our free version, they would never have considered us a viable vendor to solve the problem. Never underestimate the power of trial and free versions of your product.

My Favorite Story

This client story has to be my favorite since it is the most outrageous. An enterprise client with a vast quantity of kiosks deployed worldwide chose to use a competing HIPS product for protection. The solution was behavioral based and had a runtime mode and a learning mode. Since the product was deployed, it was left in learning mode building a profile of “acceptable” behavior. This was in the early years of machine learning technology. This process had been going on for months. Well, in that time period, a massive worm wreaked havoc on many major corporations for millions of dollars in damages. This client was no exception.

So, the first reaction of the client was to put the kiosks’ HIPs product into protection runtime mode to stop the threat. Unfortunately, they were too late, and all of the kiosks were infected. So, they left the agent in runtime mode and began patching and disinfecting the systems. After a few thousand machines, they found that the kiosks became re-infected with the worm.

In simple terms, the machine learning-based HIPS product “observed” that the behavior of the worm and profiled it as acceptable communication for the device. When they tried to patch the system and remove the worm, the HIPS product rolled off the patch since it was never “learned” as an acceptable change and reintroduced the files and runtime of the worm, since it was acceptable. The HIPS product kept re-infecting the systems!

At this time, the client was clearly getting frustrated to the point of legal action. Their business was interrupted and operations actually ground to a halt. Thank goodness this was not my product. They placed the system back in learning mode, patched the system, and retrained the system that the new behavior was truly the correct one. They needed to correct every system before they could proceed and when they finally finished went back to runtime protection mode. For some unknown reason, the new profile did not take, and every machine was rolled back to the infected state once again. Needless to say, the solution was uninstalled and never used again.

Lessons Learned:

• Machine learning can learn bad behavior.

• Automated actions can be just as bad as the original threat.

How Many Class B Networks?

While working as a systems engineer, my account manager and I did some heavy lifting in Canada. We had an early prototype of our appliance and sent it to a trusted client for his opinion, and possible upsell opportunity over software alone. A few weeks earlier, he received the appliance, configured it, and began scanning his desktop environment. Previously, they only scanned servers. A few days before the visit, we had a debriefing, and it was ugly. Scans he started weeks early were still running, and he had no results or reports from anything he had tried. The user was very familiar with our software solutions so we could surmise that there was a major problem with the new appliance. He was only trying to run a scan of one floor of his building, and only 100 devices are active per floor. The network itself was a little older too, only 10Mbps half-duplex. This should not have made too much of a difference.

We arrived onsite (I always love Canadian customs after frequent trips back and forth over the border) and began to work. We reviewed the scan settings, jobs, and finally address groups. Immediately, we identified the problem. Each floor above his office building only had 100 active devices, but each floor was segmented into its own Class B network. The scanner was trying to reach all 65k+ devices to determine which 100 were active. And what was worse is each of these devices randomly scattered throughout the range, and no one had a list of the addresses or names to build a concise list for scanning. So, the only choice was to try and scan everything. There was nothing wrong with the appliances, but it was determined with the settings required, it would take 30+ days to find the devices and let the scanner timeout on all the other addresses. It would be faster to visit each device using SneakerNet and write down the IP address rather than trying to scan for them using an address group and scan range. Why in the world the network engineers configured this building this way is beyond my comprehension. Needless to say, they created a management nightmare that normal tools could never accommodate.

In the end, we did a manual inventory and built new address groups. The appliance worked flawlessly after that and ever since.

Lessons Learned:

• Blindly scanning large address ranges takes time. It can take lots of time if you are scanning with all ports and regardless of an ICMP response. You need to wait for everything to timeout before you can proceed to the next address. Even just relying on ping sweeps to determine active devices can take a long time.

• Always check your scan settings and address groups. Long scan times can be due to incorrect options or misconfigured address groups.

• Slow networks restrict the number of targets you can scan simultaneously and increase you scan time as well.

• When scanning a distinct list of targets, you can never identify rogue devices.

The Blog from Hell

One of the new initiatives by the company was to start a blog. We identified who the writers would be and hired a consultant to teach us how to write good blog articles and set up a regimented schedule for delivery. At first, I thought writing a blog would be simple. Little did I know. The consultant recommended using hyperlinks when possible, including things like Top N recommendations, and to be controversial in order to simulate conversation. To that end, each writer submitted a sample, and the consultant critiqued it and added his own spin much like an editor and to even a greater extent, a ghostwriter. This appeared to be great approach at first, but the consultant was an expert blog writer, not an expert in security.

One of my first “sample” blogs discussed penetration testing and discussed the legality of it. Our consultant and not-so-expert security ghostwriter changed the first sentence to say, “I think penetration testing should be illegal.” Without sending the blog around for approval, it was posted and caused an absolute uproar for our new site within the community. For a few days, I received a wide variety of phone calls and outrageous statements about how ludicrous this comment was. And they were right. I ended up pulling the blog since it was altered, and writing a new one that stated, “what I really meant to say…”

The consultant quit out of embarrassment, and we now have a review procedure for all blog postings to make sure this never happens again. And, no more ghostwriters.

Lessons Learned:

• Before publishing anything public, have someone else check your work.

• If you use a ghostwriter, make sure they have enough expertise in your field to write about the topic. For me personally, I have never used one again, and everything is original.

• Consultants know that piece of the puzzle very well; do not assume they know the rest of your business well enough to be an expert.

• When something gets published to the Web, it is there forever. Even if you remove the page. Someone probably indexed it, made a cached copy, or copied it, and it can never be completely erased. Someone can find it if he or she wants to. Including my unapproved blog post.

Nice Portal, Baby

In a former life (previous job) my company had developed a brand-new portal technology. This was way before Microsoft SharePoint was even a product. We went on an aggressive marketing campaign for the product, and it started to get a really good reception for analysts and press.

For a tradeshow in Vegas, around the late 1990s, we had several thousand t-shirts made with the logo “Nice Portal, Baby.” Marketing thought this was brilliant and a great tagline to advertise the new release. Little did they know the choice of words was actually inflammatory in British English. The word “Portal” also means “Vagina” in the British dictionary. I will leave it to you to do the word substitution and see what the t-shirt really meant to our overseas guests.

Needless to say, we never gave out one t-shirt and threw out almost every single one. Their brilliant idea was not so politically correct after all. By the way, some of us snagged a few as souvenirs after all the laughing and explaining was over to management.

Lessons Learned:

• When doing outboard marketing, consider your choice of words carefully. Definitions in other languages or even dialects can mean different things.

• A pun on words can be misunderstood; vet out all the possible variations before releasing.

• Know your audience. If the clients were only American, this would never have been a problem.

• Mistakes like this are costly. Like other examples in this book, try to have someone else check your work.

Online Banking

During the .com bubble, my sales manager and I engaged in a bank that performed all transactions online. They had no brick-and-mortar locations and offered higher interest rates on savings accounts compared to everyone else since their only overhead was the corporate office, mail room, and data center. The business model sounded great, and many physical security problems were a moot point since no physical locations ever existed. Depositing checks was done through the mail and getting cash was through any participating ATM.

Early in the sales cycle, we identified that vulnerability assessment was key since all of their work was done through information technology. There were no manual procedures since the primary presence was through the Web. This was our business, so this seemed like a natural fit.

Now at this time, PCI DSS did not exist. The client was only concerned with things like worms and rightfully so. This was the biggest threat at the time. When we did an initial scan of the environment, we found all of the modern systems of the time (Windows 2000) and problems across the board ranging from null session, blank and default passwords, and anonymous shares. The client did not care since these were all behind the firewall and only the forward-facing web servers, the front doors to the bank, were a concern for threats.

Well, after our pilot, the primary contact went dark. His boss went dark. We could not find anyone willing to speak with us regarding the pilot or if they wanted to license the solution. We later learned both individuals left the Internet bank for unspecified reasons. Based on the feedback we did receive from the new security officer months later, was that both individuals participated in illegal activities and were terminated. What they were, we never found out. It was a closed, lost opportunity. My sales manager and I think they used our data to rob the bank, based on the buzz in the community. We never found out for certain.

Lessons Learned:

• Vulnerability assessment data is very sensitive information and if in the wrong hands, can be used against an organization in the worst ways possible.

• Even the most trivial of critical threats can be a real risk to the business. If you do not understand why they are critical, research the problem. Do not dismiss them based on other mitigation strategies like a firewall.

• When a client goes dark and does not communicate anymore, you have a problem. It is generally never good.

• Bank robbers will go after the money any way possible. Brick and mortar or electronic. In today’s world, a cyber attack is preferred if it can go on undetected for long periods of time.

Lies

One of my competitors just lies. You may have vendors or competitors that do too. They hand out competitive analysis documents of their products versus ours, and they are now dated and years old. In fact, when they were published, they were grossly inaccurate and aged rapidly as the solution was actively being developed.

The problem with these documents is the unwarranted defensive it places us vendors when a potential client receives them during a sales cycles. A salesperson’s initial response is to formulate a rebuttal to each statement. It is our belief that once you do that, you have already lost because you are explaining yourself and playing directly into the competitor’s game, regardless of how accurate the document is.

After trial and error, the only way to respond is the high road. The highest road possible. You do not answer the document. You explain that you have seen this before and this is standard tactic to win the mindset of clients and that it is dated and inaccurate, even when it was released. You basically play against the credibility of the competitor and make them question every fact they state that they do better.

Here is a simple example. The competitor stated that they do not need remote registry access in order to perform a credentialed scan. Lie. There is no way to inspect a remote registry on a windows host without the remote registry service turned on (This was before WMI remote enablement). They stated to the prospect that we needed it, and they did not. Hence, their scans were more secure. The truth was because our prerequisites document stated it as a requirement and theirs did not. So, in lieu of coming clean on the oversight in their documentation, they spun it into a competitive advantage that was laden with errors.

So, when we took the high road in explaining the quality and accuracy of statements such as this, we built a strong relationship and replaced their fear, uncertainty, and doubt (FUD) with our own, based on facts. Ultimately, we won the deal.

Lessons Learned:

• If you lie to the client, you will get caught.

• If you misstate features and requirements, you will get caught and have to explain it later. If it is after the sales cycle is complete, you may destroy your credibility. If it is before, you can easily lose the deal.

• If you choose to build competitive comparison sheets, be prepared to dedicate resources regularly to keep them up to date. Even after a few months, a single release can change the facts and make the document, and its claims, inaccurate.

• If presented with a competitive document, never answer it line by line. It is not an RFP. Take the high road and explain why this approach is flawed. This can be based on business, technical, or even product. Never let an organization compare apples to orange in these scenarios.

Speaking of Comparisons

A regular mistake my sales team makes is requesting comparison documentation and feature tables not with competing products but with their own. This is not to say that comparing your own products is bad, but there is a time and place for this type of documentation and a place when you should never do so.

First, consider a table on a website comparing a free version, to a professional version, and ultimately the enterprise version. This allows you to upsell features and functions of the same product, using different releases, to larger or more experienced clients. This in general works well when you can justify the cost difference between features and functions.

Second, this does not work well when comparing two different products in your portfolio that have overlapping functionality. You have actually turned your own sales cycle into a competition between two products you are trying to sell, one with more or less functionality than the other but in a completely different family.

Consider two products that can do web application vulnerability assessment. One only does web app scanning really well, and the other only does basics with additional scanning functionality for operating systems, applications, and databases. If you compare the two, you highlight the shortcomings of one product (why doesn't it do that) and enforce that your technology is not integrated and lacks a common vision to solve a problem. Why else would you have two different products that do the same thing?

Comparisons are a great way to upsell your technology but a poor way to compare your own family of products. When you have this situation yourself, stick with one product and lead with it. Giving comparisons will just confuse the sales cycle and make you compete against yourself.

Lessons Learned:

• Use comparison documentation correctly to upsell your own products. Not to compare them side by side, literally.

• Avoid pitching two of your own products to clients to solve the same problem. You end up just competing against yourself and confusing the client.

• If you have to explain the differences between two (or more) products that do the same thing or have overlapping features, your strategy for them as solutions is flawed.

Getting Your Facts Straight

Early in my career, I had a salesperson who would embellish facts in order to make the sales cycle slant in our favor. They were not blatant lies but light exaggerations of facts in order to make us look more favorable. This always created problems, especially when both of us would present to a prospect in the same meeting.

This problem led to the angriest I have ever gotten at work and happened after one of these meetings. My salesperson embellished a statistic for an SLA, and when I presented later, I inadvertently contradicted him. He accelerated the time frame (of course), and I stated our contractual requirements.

The client did not appear to notice the difference (we won the business), but the car ride afterward was a bloodbath of angry words and accusations.

After the fighting and yelling was over, we agreed on one simple principle: to listen to the other person’s presentation, regardless of who goes first and use their statistics in the rest of the conversations with the prospect. This was incredibly important since sometimes I would go first and present a fact and he would contradict and vice versa. So, getting our facts straight from beginning to end of a pitch was incredibly important.

Lessons Learned:

• Listen to presentations from your peers before you go on to the same audience. Use their facts, emphasize their message, and stay consistent.

• If a peer lies or embellishes a fact, never contradict them in front of a client. Always do it later, in private. You might be surprised that you are actually wrong.

• Yelling at a peer never solves any problem. Keep your cool. This may be obvious, but I have seen it happen too many times, especially to subordinates.

• If you embellish a fact, be prepared to back it up. SLAs can easily be jaded based on collected data, but other facts are not so manipulatable.

Dependencies

One of the biggest “gotchas” for any pilot is system hardware and software dependencies and prerequisites. Most vendors publish an extensive list of what a host system or VM should have in order to support the solution. The problem is that most clients never review the list and provide hardware that is grossly underpowered for the task.

Some manufacturers have resorted to automatic checkers to verify the requirements prior to installation, and others will enter a pilot with appliances that are fully preconfigured in order to avoid these problems, delays, and potentially poor user experiences.

So, this is where our story begins. The current documented requirements recommend a minimum of 4GB of RAM and specifics for the configuration including an installation of MS SQL.

Well, the prospect wanted to engage in a software pilot and decided to install the solution in a VM with only 1GB of RAM and SQL Express. They did not have any spare hardware, and their virtual hosting environment could not support any additional resources.

Of course, the pilot was a disaster. The solution was sluggish, features just did not work, and the user experience was awful. The prospect was still very interested in the solution, acknowledged the shortcomings in their environment, and requested we send a physical appliance. A few weeks later a box arrived in the office.

After setting up the system and doing initial testing, it was clear that we had additional problems. Even though we used an appliance, other environmental dependencies were not met, and certain other features just did not work. This included very old versions of Internet Explorer still in use that tried to access the management console on the appliance and existing installations of WSUS.

Needless to stay, the appliance did better, but we only overcame about half the problems. The client, nor the sales team, bothered to finish reading the prerequisites list and verifying all the requirements for host prerequisites to client dependencies.

Well, the client begrudgingly upgraded a few hosts and began to use the pilot solution with limited success. At this time, they were becoming bitter from all of the back and forth and just wanted a solution that plugged in and worked. None of which was true yet. Every problem from requiring Internet access to license the solution to spam filters blocking license keys was experienced with this client and proved that a prerequisites checklist alone is not enough.

So, after months of working with this client, they did not make a decision to commit to our solution or more importantly anyone else's because none will work out of the box in their environment.

Lessons Learned:

• Before any pilot or post sales installation, always verify the prerequisites.

• If you encounter basic problems like Internet access, your competitors will probably have the same issues.

• Even if your organization has simplified the installation using a VM, software-based checkers, or even an appliance, other dependencies can cause issues. It is important to complete the prerequisites checklist and if new issues arise, fully document them for future clients.

• If a client insists on using older and outdated versions and technology, make sure you can support them, especially for the length of the contract. They will resist upgrades and potentially could make you support older versions that can drain support and QA resources. For example, do you still support Windows 2008 or even Windows 2003?