A little over seven years ago, I wrote the first edition to this book, and to be honest, I could have done better. So, when Apress reached out to me to refresh this book, I was elated – elated because not only do I get the opportunity to do better but also because in the space between the first and the second edition, I had the opportunity of taking a fantastic journey. It’s a journey that had many highs and lows – one part where I led a team of software builders and one where I led a team of software breakers. I will give you an insight into how it feels to build and secure a product in a company which went from 40 people to over 3500 people seemingly overnight – a company that, as of writing this book, has a valuation of ten billion US $.
Before I take you on this journey, I would like to thank Steve Anglin at Apress who gave me that second chance at doing better and who still believed in me enough to reach out to. Together with me on this book is Mark Powers who I have worked with closely in making sure all parts of this book are good to go. Also, I’d like to thank all the folks on the team who I will not interact with directly, but who I know are there making sure this book looks its best. A big thank you to all of you.
The Startup Landscape
Most folks who have not been living under a rock are keenly aware of how the “startup ecosystem” has witnessed meteoric growth. I attribute this growth to the VCs and investors that have been pumping in hundreds of millions of dollars into round after round of fundraising. With more capital being available, it seemed like more startups popped up out of thin air. At the center of each startup was almost always an app. Be it an iOS or Android app, it existed and was mostly touted as a solution to a problem at your fingertips. Now this was not happening only in the United States, although it really took root there. Word of startups that built apps spread around the globe. Headlines proclaiming what seemed like colossal sums of money being raised by these startups made everyone everywhere sit up and take note. Asia was no exception. It saw the success that the startups in the United States had and raced to keep pace. The startups were different, though. Asia (excluding China – because China’s startup ecosystem is considerably different) had a different set of challenges, and thus the startups that were born there had a unique set of operating conditions that made it unique to Asia alone. Some challenges were a lack of stable infrastructure, poor handset quality and thus handset performance, and a more reserved culture when it came to trust, payments, and app usage.
Through it all, however, one key datapoint has remained a crucial success factor for an app waving startup: the user base, or the number of users of that product or app. For startups to attract large funding rounds, it had to show growth. One key metric that defined growth was weirdly not revenue. Instead, it was the number of users that interacted with the app. The more users you had, the more valuable you could become in the eyes of VCs. The formula was simple: gain more users, gain the attention of the bigger VCs that would fund you well. The startup founders did just that; they went after the “critical mass” of users to help kick off the success of their creation. With this, the flow of VC money poured into Asia and even gave rise to VCs grown from within Asia. In an article1 written by the Nikkei Asian Review in October of 2019, the numbers for Asian-focused VC funds sat at 323 billion US $ under management, while the numbers for US-focused VC funds were at 397 billion US $. The numbers were taken at the end of 2018, and the article projected an eventual overshadowing of US VC funds by the ones in Asia.
So, what does this all mean? Well, the currency of startups these days is not actual currency. No. The majority of these startups are not even breaking even. The currency of startups in present day is users – users, glorious users. Users with their personal information, credit card numbers, addresses, email addresses, and unencrypted passwords or password hashes, it’s quite the buffet for anyone that wishes to feast. The bigger the funding and the company, the more upmarket your buffet. I would even go so far as to say if you are tracking your attacks on your infrastructure regularly and see a rise to a new, higher norm, chances are you’re making it big in the startup industry. Pat yourself on the back and go get yourself a CISO if you haven’t one already.
Between Two Books
A lot has transpired between the two editions of this book. Startups were formed, users were gained, users were hacked, and users were lost. Let’s take a quick look at the security landscape between the years of 2013 and 2019. One thing that you may notice when you go back looking at breaches is that apart from news of Android malware, everything else seems like a hacking attempt. Let’s look at the malware first.
What Is Malware?
Malware is defined as any piece of malicious software that lives on a user’s computer or smartphone whose sole task is to cause destruction of data, steal personal information, or gain access to system resources in order to gain full control of the device it is on. Malware is written with the sole intent of causing harm; and usually, malware authors will write the malware to target a specific weakness in an operating system or platform. Often, the malware author will want to maximize the spread of her malware and will look to implement a mechanism where his software can copy itself to other, similar devices.
Now, previously, I used the word “lives” on a user’s computer or smartphone. Let’s dissect how the malware gets to live on the device in the first place. Obviously before the malware starts living on the phone, it has to be introduced. To explore further, let’s look at a piece of Android malware that has managed to steal over a million Google accounts. It’s named Gooligan. Gooligan was first discovered as a mobile malware campaign that was not as malicious as it became.
The user interface for SnapPea
Now SnapPea, it was found, would try to root your Android device with 12 different exploits. Check Point has remarked that it was the most amount of Android root attempts that they had witnessed in the wild. Two of the exploits were found to be the vRoot [CVE-2013-6282] exploit and the TowelRoot [CVE-2014-3153] exploit. These two are the only exploits that have CVE IDs in the database. When the rooting of the device was complete, the malware would install several other malicious apps that would further install subsequent ad-laden apps. The malware would then connect to a central server and await further instructions. It seemed like the primary purpose of this malware was to install other adware apps that made all infected devices connect to advertising sites and make it appear like legitimate users were viewing the ads. More views meant more payment for the app owners, and so greater infections yielded greater profits.
How the malware made its way onto a device
Subsequent iterations of this malware, which then became known as Gooligan , used similar rooting mechanisms, but had far more sinister goals in mind. After the original malware authors released their adware/malware campaign with SnapPea in 2015, they retreated and disappeared not to be heard from again until June of 2016 with Gooligan. Once again, apps that were available on third-party app stores carried the malicious payload. The apps were marketed as free versions of popular paid apps. So essentially, the draw this time was that you’re getting a free version of an app which you would normally have to pay for – piracy.
How Gooligan worked
Launching Attacks via Phones
Barring a situation similar to Gooligan earlier, it is important to keep in mind that attacking a phone gives you far less reward for your effort. The logistics of attacking a phone directly in that manner are beyond the reach of the solo hacker. Typically, organized malware attacks such as Gooligan are the work of larger, organized groups.
My API URL is an unknown, unpublished one
My API can only be accessed by our mobile app that was built in-house
These assumptions are fairly normal ones to have in general if you have not had to think about the security of the overall product. Lucky is the paranoid developer that considers his code unleashed into hostile waters the minute he publishes it. The two preceding assumptions usually will lead to more lax security that can take on the form of sending too much data between phone and server, relying on the phone to carry the load of authentication, offloading some processing onto the phone, or storing sensitive data on the phone. The assumptions are wrong. Imagining for even a second that your data in transit between the app and the back end is safe from prying eyes is an illusion. I will show you later on in this book how to look through the data traversing between phone and server.
You may have seen me use the app and product earlier. By app, I mean just the application that is running on your smartphone. By product, I mean both the app and the back-end component which contains the API that the app will talk to.
OK, let’s keep in mind that some developers may make the assumptions listed earlier and proceed to build and launch a product. We now have a product that makes assumptions about security. The product is released and the public loves it. Your startup begins to build a group of loyal users. Joy of joys you even begin to see repeat users when you do your cohort analysis! You begin to attract the attention of investors, and then the media starts writing about you. Let’s hit the pause button here. Similar to the Hollywood movie effects, our actors all come to a halt with a sound that a record player makes when you physically stop the record with your finger.
What you have here is a pivotal moment when things can go colossally wrong, or you hire that CISO and they yell at you for getting this far without paying attention to security. Since we’re tracking the security incidents between 2013 and 2019, let’s go with colossally wrong. Let’s unpause our scene.
Because of all the attention you have been getting from users, investors, and media, you also begin to attract the attention of bored or motivated hackers. A bored hacker who has a little too much time on his hand scrolls down past the news article about your product, then scrolls back up again. He thinks, “Hmm, I wonder what those guys are sending back and forth between back end and the app….” He downloads your app onto his rooted Android phone, plugs it into his laptop, then fires up his favorite set of tools on that laptop, and within minutes he begins to see all the traffic you send back and forth. He smiles, “Someone decided to send phone and credit card numbers in requests that only needed to send a name. Oh, that was particularly lazy of you to not filter out those bits of data.” His smile widens as he whispers, “I have you now.” End scene.
My gigantic ego would immediately associate myself with the superior skilled, elite hacker in the preceding story, if not for one thing: that was a true story of me when I tested an app in September of 2019. Yes, yes. As you will soon see, I have a flare for the dramatic. If you ask my daughter, she would even say melodramatic. Moving on to my theory, I firmly believe that a major portion of breaches involving a loss of large quantities of personal data stems from a weakness found in an app – a weakness similar to what we discussed earlier. The underlying assumption that all is well in the security of the product ecosystem can give rise to some stressful times that can make or break your business.
So, what truly happened between the first and the second edition of this book? Not a whole lot of change sadly. I’ve been testing app security as recently as two weeks before I started writing this book, and I still find the same attack vectors that I found over six years ago. Yes, this is an incredible unhelpful thing to say and probably goes on to even alienate some of the developers out there. I’m from the tough love school, and so, I feel that the best way to think about security, especially in a startup worth millions of dollars, is that you only get one chance. This is another reason why I am taking a crack at writing another book, so that we can all go through some of my experiences together and try to come out of it with better security. Right? Great! Let’s now move on to some of that journey I’ve been promising you earlier in this chapter. Let’s first start with some of my lows.
Hello, I’m Your CTO
As I am apt to tell whoever I am speaking with or addressing, I was extremely fortunate enough to join a startup in Southeast Asia, specifically Indonesia, called Gojek. Now if you haven’t heard of Gojek, that’s fine. Let me give you a brief idea of what we set out to do. I used to tell folks that were unfamiliar with Gojek that we were an Uber for the two wheelers aka motorbikes. To address your still burning questions, we must turn to the transportation infrastructure in Indonesia. If you ask an average Indonesian living in the capital city of Jakarta, they would say that the transportation system, regardless of what mode of transport you took, involved sitting stationary for large chunks of time. The word in Indonesian is called macet (mah-chet) which means “traffic jam.” Macet was so much a part of the vernacular that a group of people you were meeting would nod sympathetically if you walked in 20 minutes late. Even if you had an actual legitimate reason or emergency for being late that didn’t involve a traffic jam, your business counterpart would look up at your disheveled appearance and offer up a polite “Macet?” before you would start your meeting. Now because of macet, four wheelers aka cars would stand no chance in making up any lost time between your two meetings for the day. What emerged was an informal group of motorbike riders that thought, “Hang on, we can zip through macet and make some cash ferrying people who were late between their two meetings for the day.” Essentially was born the culture of the motorbike taxi. It isn’t entirely new and is usually found in cities that were developing fast and didn’t have the infrastructure to cope. Thailand has one called the motorcy. In Indonesia, they called themselves ojek. Ojeks were great! They could get you back and forth fairly quickly but came with a slight problem: price haggling. If you wanted to get around, you would find an ojek asleep on his bike on a street corner, then tell him where you needed to go and why you were refusing to pay his ridiculous asking price. You can see where this is going, can’t you?
Two friends of mine, Nadiem and Kevin, thought, “Hey, let’s make an app for these ojeks! People don’t like the fact that they have to haggle all the time so let’s make that process simpler and build an app to hail and ride ojeks.” Although Gojek was founded in 2010 as a rudimentary web page with a call center acting as an ojek dispatch, in 2015 they launched a mobile app for it. A rebirth if you will. The guys offered me to be one of the cofounders and picked me to be their first CTO, and you might think I jumped at the chance. Well, you would be wrong. I was skeptical to say the least when I heard the concept. I wondered where all the technology was that needed a CTO. But Nadiem, in his usual persuasive manner, asked me to try it for three months, and if I didn’t like it, then we could part ways. I ended up staying for almost four years. (One thing you should probably know is that a Gojek year was roughly equivalent to 3 normal years. I am deadly serious.) We went from a small startup of about 40 people in 2015 to one that is well over 3500 people today. An app that started with three core features then, transitioning to a super app with more than 20 features now. A company with minimal funding then, now valued at 10 billion dollars.
As the CTO, I was tasked with keeping the product running, building the product, building a team, and ensuring all our technology needs were met. To say that it was challenging would be an understatement – mostly because a lot of the things I had to do, I had to learn on the job. No prior role of mine prepared me for the complexity and stress that came with being the CTO of this new hot startup out of Southeast Asia. On most days, I felt like I had failed. We had so much growth, so few developers, and so many system failures that caused downtime. Our tech team was five people in those early days, and I would rack my brain trying to find and hire more developers while simultaneously helping with coding the back end, optimizing APIs and infrastructure maintenance. We went on like this for some time, and we slowly improved. Our product got better, we launched features fast, and we got funded. I was out of my element though. I was struggling because the role for me was one that I was diametrically opposed to: here I was building things when all I was used to doing was breaking things for over 15 years. I pushed on and learned a lot. I learned about building teams, about communication, and about accepting that good enough is good for now. I had never seen a scale or speed like that in my life, and it felt quite lonely because there was no one around that I could talk to about it. Yet I was happy, I was doing something I thoroughly enjoyed, I was challenged on a daily basis, and I didn’t have time to think or take a break: bliss.
We were chugging along as usual balancing the product features, performance, and growth every day for roughly one year when I came to the realization that I was probably in over my head. While I knew that I could handle the role of CTO given enough time to catch up, this particular business was ruthless and unforgiving in the time that it gave you. I also knew that with our continued trajectory, we would desperately need to get a handle on security. Early on I had started to build a small security team that would sit there and hack our product to uncover any security flaws that could affect our users’ privacy or affect the business negatively, but I knew that I would have to increase the size of this team and provide leadership in that area. With that, I had separate conversations with both Nadiem and Kevin and transitioned my role to be the CISO.
This was a bit of a low point in my time at Gojek because I was in a learning mode and doing something I had not done before, and being challenged at it was immensely satisfying. But in reality, there was a desperate need as we grew to make sure that security was tight, and I knew, given my background, I had to step up and deliver for the good of the company. As time passed, we even coined the phrase to signify important moments like this: “It’s not about you.”
The next low point was some time after I had transitioned from the CTO role to the CISO role; we received several emails from both well-meaning individuals and glory and profit-seeking individuals alike that told us they had successfully hacked our product and gave details on what they had uncovered. To strengthen my theory that I mentioned earlier, it turned out that they had breached our API back end by hacking our Android app. They discovered the communications patterns between the app and the back end and were able to compromise our product including collecting our customer data. In the past, I have also been able to report flaws in products to companies including BlackBerry, and I always appreciated a company that reacted and responded fast. Still, nothing quite prepares you to receive an email like that – one that says that someone out there has managed to successfully compromise your product and offers you proof. I felt my stomach lurch and a wave of nausea washed over me. Here I was, part of an organization that was on the receiving end of the hacking for a change; usually I was the one hacking others. Nevertheless, I took a deep breath, notified our leadership team, and took this to the tech team so that it could be rectified quickly.
Getting hacked is truly a terrible feeling. Having it unfold in public while hyped by the media for sensationalism makes that feeling worse. I empathize with anyone that has been through a scenario like that. It is a lesson that is swift and often one that you will not forget. It can make or break a company or a career. It leaves you feeling vulnerable and exposed. It is a feeling that I want to help you avoid. It is a mistake I want to prevent you from making. Therefore, I’m going to pack as much of this book as possible with real-world instances from which I learned some important lessons. Now there will most likely be different ways of handling a particular situation, but I will tell you how I did it. In the first half of this book, I will discuss and document many of the areas you should be looking at as a builder of software when you are part of a startup. I will outline some techniques and tips that you can consider when you have to build something at scale. Your teams may be large and diverse, your communications may not be as streamlined as you would like, but I will talk about some practical mechanisms that you can adopt to improve your security as a builder. Not all of it will be a narrative as my stories in this chapter; this is a technical book after all. Yet, I may occasionally wax poetic and regale you with my tales of the time I spent at this startup that is close to my heart, named Gojek. As for the second half of this book, I dedicate this to my people, the breakers…
Hello, I’m Your CISO
And I will help you to break apps so you can be better informed about how to fix them and avoid feeling what I felt when I got hacked.
A startup CISO’s job is still fairly new. Acceptance has just begun because the number of articles I read on the topic is increasing steadily. Where to put your CISO, who should he report to, and what should his job description be are all discussed more frequently. Yet, the answers you will get will vary by and large by the person answering it and his agenda. I had plenty of discussions when I started in my role as CISO at Gojek – endless debate on whether I should report to the CTO or to the CFO or even the CEO. As you may have guessed, we had moved on a little from our startup roots at Gojek and were now emerging into a more structured operating model.
Let’s tackle these questions in this section because I think they are important. Let’s try to look at the role of the CISO from a few different perspectives. Let’s start with the CEO and founder.
Reporting to the CEO
The founder and CEO of a startup does everything in the beginning. No task is too small; no job is beneath him. A founder and CEO will agonize over the longevity and survival of his idea and his company. How can I hire the best? How can I accelerate faster? How do I stay ahead of my competition? Where can I find money to pay my employees and keep the business running? These are some of the many questions that likely bounce around in his head. If we try to break down the role of a founder and CEO, it is one of removing friction at all costs. All his tasks and his purpose in the company are to remove friction and enable growth, sustainability, and survival of the idea, vision, and company. Thus, to report to a CEO, the CISO must be willing to be an enabler of growth, of technology, and of features. A CISO reporting to a CEO cannot be too restrictive, but at the same time, a CISO reporting to a CEO has to be an educator. He has to be someone that can demonstrably show the risks involved when setting out on a particular strategy. Communication is key in this relationship. More than that, it is important to be able to understand your CEO and his goals so that you can effectively highlight the risks you may see on the path to these goals. For example, if your CEO is unaware of fraud that is taking place due to a flaw in the logic of your product, then learn to present the problem in terms that he would respond to. If he is a number or a data person, then quantify the fraud in the form of either a dollar value loss or a number of users lost datapoint. If he is more of a socially inclined person, then present how the fraud could cause a loss of productivity for the large number of users of your product. Your CEO is highly likely to be someone that has a bigger vision so understand that and try to adapt your narrative to show how what you propose will help drive toward that bigger goal.
Reporting to the CFO
This relationship, while rare, can still be one that you find yourself in. In this dynamic, the role of the CISO would most likely be weighted more onto the regulatory and compliance side of the spectrum. Technology security, such as red teaming and hacking your own product, may come second to ensuring that sufficient and sensible security policies and operating guidelines are in place. A CISO in this relationship may find that he has more leverage over individual teams or departments. The mantra of “do it this way or we are not in compliance” may follow this CISO and with good reason. Financial startups or startups that answer to regulators such as central banks have a strict set of guidelines on security that the startup must comply with. Often, noncompliance can lead to large fines or even a loss of an operating license that could bring the business to a crashing halt.
Reporting to the CTO
Don’t.
I kid, I kid. This relationship is one that is mostly fraught with the greatest conflict of interest. The CTO’s role in a startup is to deliver a feature-rich product that iterates fast and is performant. Finding a flaw or selecting a strategy that is more secure may directly affect the CTO’s ability to do his job. We can all theorize about carving out a small team to fix any flaws discovered by the security team, but when it’s crunch time and engineers are limited and the investors are breathing down your neck and the competition is seemingly moving faster, it becomes near impossible to slow down and address security flaws. This is a significant problem especially if the CTO isn’t well versed in or does not come with a security mindset. Should a CISO find himself in this relationship, then pre-agreeing with the entire leadership team about a structure of transparency should be something to consider. The worst position for a startup to be in is one where none of the security flaws discovered either in-house or externally are being brought to the leadership’s attention. While there may be some short-term respite for the tech team, it rarely bodes well for the entire organization in the long run.
In my role as the CISO, I reported to Nadiem, the CEO. I structured my team in three broad areas of specialization: the Red Team, who would mount the offense on our product and infrastructure; the Blue Team, who would work to actively secure the infrastructure; and the Compliance Team, who would build, maintain, and disseminate policies and guidelines. The Compliance Team would also work together with the regulators on any licensing requirements that were necessary.
Reviewing What Gets Published
Being a fast-growing startup, one of the bigger challenges we faced as a security team was trying to figure out what products existed and which ones were launched newly. Essentially, this is inventorying, because you can’t secure what you don’t know about. Working closely with the infrastructure team, we took control of the process of adding a new set of APIs to our load balancer. This created a sort of gate that allowed us to question the requesting team whether they had conducted a preliminary penetration test on their system. If they hadn’t, then we would not proceed with the release of the APIs until they did. Since we had announced this and continued to communicate this fact, it was rare that we encountered a team that requested an API addition having not completed a penetration test.
Another such “gate” scenario you can consider is to place a source code security review service as part of your software build process. Usually, tech teams will have build processes known as CI/CD or continuous integration/continuous delivery . Essentially, this boils down to having several teams contributing code that is merged to one main development branch and having shorter development cycles so that software can be deployed or delivered faster. Sometimes we would release software three or four times a day. Things like new features or bugfixes could go out very fast thanks to this process. One prerequisite to deliver the software was that the “build should pass.” This means that prior to a release, all the source code that has been contributed is built. Once built, the software can be released. The build process will have a set of tests that it runs to ensure the software does what it is supposed to. For example, the component that added a food item to your order was tested to see if when the item was added, the correct item was added, and that the correct quantity was added. If any of the tests would fail, especially on the critical tests, then the entire build process would stop with a failure notice. This meant that the software could not be released until the failures or bugs were addressed. By adding the source code review system to the path of the build process, your security team can ensure that any code committed to the main branch passes secure coding practices (at least as effective as the product used for source code review). If the source code reviewer found a flaw or an instance of insecure coding, then it would trigger a build failure. Thus, it would be possible to prevent insecure software from being released. Now keep in mind that this would just catch the low hanging fruit of the insecure source code. Although source code checkers are evolving, they are still not up to a level where they can prevent the majority of attacks.
Did I Just Waste My Time Reading All This?
I don’t think so. Yes, it is a lot of narrative and very little to do with Android apps or anything deeply technical, but I think it sets the context for this book. I feel that by contextualizing some of the technical topics that come later in this book, it can add a little bit more value as to the “why” of doing things and doing things a certain way.
Startups, especially technology product startups, are here to stay. The mobile device is here to stay. It would be very interesting to get a glimpse into how all this will further evolve, but for now, looking at the tech and security through a startup lens adds a lot of value in my opinion. As the chapters unfold and go from defensive to offensive security, having this context in mind should serve you well. Just for the heck of it though, the offensive chapters should be fun to read without context either as I will write them as a handy reference guide that you can directly flip to, get setup, and go from there.
With all that being said, let’s head over to our next chapter and review some of what we learned previously with regard to secure software development principles. See you there!