This book began with a riddle. We wanted to know why a company based in a redbrick house in the English town of Rochdale was lending £2 million to two-time European champion FC Porto to sign a striker. The company listed its address as 35 Princess Street, a scruffy part of town neighbouring abandoned buildings with bricked-up windows, an advice centre for the homeless and a builder's yard. CCTV cameras pointed at the front door.

It was 2010 and the Portuguese club had recently reached the last 16 of Europe's Champions League with a squad that included Radamel Falcão, Givanildo Vieira de Sousa – better known by his nickname of Hulk – and Nicolás Otamendi. Over the next few years these three players, from Colombia, Brazil and Argentina, would subsequently fetch transfer fees totalling more than €150 million and play at some of Europe's most illustrious stadiums, including Manchester United's Old Trafford and Manchester City's Etihad Stadium, a dozen or so miles away from that ordinary Rochdale house.

As the new season got underway, Porto used the £2 million loan to help finance the signing of a striker to provide cover for Falcão and Hulk. Walter da Silva, who had been raised by his mother in a Brazilian slum, was very much an unproven talent, even if he had made his debut in Brazil's Under-20 national team a year earlier.

A second company, based in London's Chancery Lane, provided another €2 million to help finance the €6 million signing. Porto's published accounts showed that this company had acquired 25% of the transfer rights of the striker. That meant it would take a quarter of any fee the striker fetched if Porto traded him to another team before his five-year contract expired.

If Da Silva went on to became as sought-after as Falcão, Hulk and Otamendi, it could turn into a lucrative investment. Although it took us three years of digging to find out who the owners of both companies were, it was clear from the start what they were doing: seeking financial returns from the football player transfer market.

Transfer fees have been part of the game since 1890, when they were introduced by the English Football Association to compensate smaller teams for losing their best players to larger clubs. Back then, there was already fierce competition among English clubs for the best talent and the manager of Preston North End, then one of the leading teams, tempted amateur players from the Scottish dockyards with money and jobs such as running a pub.

Transfer fees took longer to catch on in Italy, Spain and Germany, but were later adopted worldwide by the sport's world ruling body, the Fédération Internationale de Football Association (FIFA). FIFA regulates the thousands of cross-border transfers that take place each year.

Over the last 20 years, the international transfer market has mushroomed sevenfold in size, into a €4 billion-a-year business. The boom has been fuelled by a steady increase in income from broadcasters and an influx of billionaire owners into the game, such as Russian oligarch Roman Abramovich and Abu Dhabi's Sheikh Mansour bin Zayed Al Nahyan, who own Chelsea and Manchester City, respectively.

It was also underpinned by something else: clubs perpetually engage in panic buying of players. In 2014, Ebru Köksal, a former Morgan Stanley banker who was general secretary of Turkish club Galatasaray, explained to a room full of football financiers in Zurich: “When the Galatasaray president has a stadium full of people chanting for him to resign, he goes to the boardroom and says: sign three players.” Even if, she added, the finance director says we can't afford to do that. “I'm not proud to say that for the last 10 years we've been carrying negative cash flows.”

Across the game's European heartland a similar theme played out. In a study of 44 teams that year, S&P Capital IQ deemed every single one – including 20-time English champion Manchester United – below investment grade (sometimes known as “junk” grade). That's because they were ploughing most of their revenue into transfer fees and player wages in an effort to appease impatient supporters, win trophies and collect more prize money and sponsorships.

Today, the player transfer merry-go-round consumes the interest of the sports sections of tabloid newspapers and football websites, as supporters hungrily consume speculation about which player might fortify their team. The frenzy reaches a climax in Europe on August 31st and January 31st, when sleep-deprived lawyers and agents criss-cross the continent on planes trying to finalize deals before the midnight trading deadline in Europe, where most money is spent. Occasionally, they carry suitcases of cash to help ensure deals go through.

Only 10% of the 13,500 international transfers in 2014 fetched a fee, but for a few hundred elite players the fees are enormous and can top $100 million. The top end of the market is so exclusive that FIFA's former transfer market compliance chief Mark Goddard described it as “like a yacht club”. There is nothing quite like it in any other sport or industry, although you could say there are some similarities with the sale of thoroughbred racehorses for millions of pounds at auction houses where Arab royalty are among the bidders.

To try and keep up with billionaire club owners like Abramovich and Sheikh Mansour, for whom money was no object, some Champions League teams had been getting finance from lenders whose identities were not immediately clear. In our search for the people behind Porto's transfer finance, our first stop was with the UK corporate register known as Companies House.

The companies in Rochdale and London's Chancery Lane each had a single named director. We contacted both of them, but neither returned our emails and letters. Between them they had been directors of more than 230 companies in Europe over the previous decade, making it likely that they were so-called nominee directors.

Such officials are hired, quite legally, for a few hundred pounds a year to sign off on company accounts. “You can form a company to do a deal, say one football transfer, and then get rid of it almost straight away and never put any information on public record,” Richard Murphy, a tax expert based in Norfolk, England, told us.

We would find out that a series of UK companies had been used to finance football transfers of players over the last 15 years. There were many more such entities around the world: in Panama, Gibraltar, Malta, Luxembourg, Jersey and the British Virgin Islands, to name a few.

Acquiring the transfer rights of players was completely within FIFA's rules at the time we started investigating the practice. It had taken hold in the 1980s in South America, where clubs received finance in return for a share in the future transfer fees of young players. FIFA had long had a laissez-faire attitude to these arrangements, provided the financiers did not take control of the careers of players. We realized that these arrangements were becoming more common in Europe with the financial crisis that took hold in 2008 as high-street banks pared back lending to many clubs.

Aside from Porto, another team that had turned to private lenders was Atlético Madrid, which had been caught short by the credit crunch. Atlético was desperately trying to keep pace with Real Madrid and Barcelona, the world's biggest teams by revenue.

Finding information about these alternative finance deals was tricky. The transfer market is cloaked in secrecy, largely because clubs do not want their rivals to know what they are spending on fees. It's rare for fees to be disclosed, and less common still for club executives to discuss with reporters how they are financing the fees. Jochen Lösch, a German sports executive in São Paulo who helped run a fund that invested in the transfer rights of players, gave us his take on the new arrangements that were replacing traditional bank loans.

He said that they were already deeply entrenched in South America and parts of Europe. “It's a bit like reading The Sun newspaper” he said, referring to the UK tabloid newspaper known for celebrity gossip. “Everyone does it but nobody admits it.” The investors did not want to be in the media spotlight that football brings. Lenders were also wary about their relationship with clubs entering the public domain.

Spanish high-street bank Bankia's partnership with Valencia football club was a case in point. Bankia's public relations department went on high alert in 2012 when the club teetered on the brink of insolvency while owing the lender 200 million euros. Having traded its best players for €95 million, the only major asset Valencia had left was its creaking Mestalla Stadium. Imagine the public backlash if Bankia had called in its debt and forced the club to sell its home.

Eventually, some club executives and lenders agreed to talk to us. They said that these private lending agreements were necessary during the credit crunch because banks had stepped back from football. The arrangements were a kind of financial hedge that allowed them to share the cost to sign players. In most other parts of the world it was a legitimate form of financing, and had been approved by stock-market regulators in Argentina and Portugal. The Spanish and Portuguese leagues endorsed the practice, as their teams were ravaged by financial meltdown.

A former English footballer in this business gave us a window into this secretive world. On a rainy day, he showed us into his ninth-floor modern office among Manchester's neo-Gothic spires. He introduced us to a sports science graduate. On his computer, the young man pulled up an eight-page file that focused on Luciano Narsingh, a winger who was born in Amsterdam.

Narsingh was coached at the Ajax youth academy that produced Johan Cruyff and Dennis Bergkamp. He did not make it into the first team, because weighing barely 60 kilos he was deemed too fragile. Now playing at PSV Eindhoven, he was starting to show that Ajax might have made a mistake in discarding him.

According to Smith's data, Narsingh was rated fourth of 80 wingers in the Dutch league, with a 69.9% score based on a variety of information such as the number of his passes that had led to a goal. That ranking made the grandson of Indian immigrants to Amsterdam a possible investment opportunity.

“This is fantasy football on steroids,” explained the former player. “We are not trying to find the next star kicking a tennis ball around in São Paulo. We are higher up the food chain.” He said that he had invested about $50 million on behalf of investors and was in discussions with a pair of quantitative analysts who have worked for Microsoft and Vodafone to come up with an algorithm to identify the players whose value was most likely to increase.

Betting on transfer fees was a potentially risky bet. The careers of young players can often come to nothing, through injury or just because they turn out not to be as good as people once thought they were. Only a fraction – typically less than 20% – of 16-year-olds in a club's youth team actually make it to the first-team squad.

For investors willing to take this risk, there were different variations on the same model. Sometimes they would buy stakes in players directly from a club. On other occasions they were passive investors who would put money into a transfer market fund like the one managed out of Manchester. A few times, wealthy individuals might even go as far as buying a small club for the purpose of speculating on the transfer market.

All these methods were legal. FIFA's regulations on the subject were distilled into two sentences in a sub-clause of the game's 39-page transfer rules. They said that the world ruling body has the right to sanction any club that allows a third party to interfere in player transfers. Otherwise, it did not have a problem with the practice: it did not publicly accuse or sanction any club for breaking the rule between 2008 and 2015.

Behind the scenes, we discovered investors who were speculating on the transfer boom, including oligarchs who had once been close to Abramovich and friends of the former Portuguese Prime Minister José Sócrates. Others included a British racehorse owner and a commodity magnate who normally traded not in athletes but in diamonds, gold and pharmaceuticals.

In South America, where the practice was most widespread, there were even people with ordinary jobs, such as waiters and taxi drivers, who had taken stakes in the careers of players.

We found investors who had profited from acquiring a stake in the future fees of star players including Cristiano Ronaldo, Neymar and James Rodriguez. Betting on that trio, we calculate, they netted a total of €15 million.

On the stairs of European ruling body UEFA's cavernous glass headquarters overlooking Lake Geneva, we told its then general secretary Gianni Infantino how companies in the UK whose ownership structure was not publicly available were being used to finance transfers. As we explained, Infantino frowned. “It should not be like this,” he said. Even if the agreements were legal and there was no suggestion of any foul play, the Swiss-Italian said the source of the finance should be clear to make sure the integrity of football was not at risk.

Only a few years earlier, in 2007, a scandal shook the Premier League when an offshore company controlled by oligarchs engineered control over the careers of Argentine players Carlos Tevez and Javier Mascherano when they joined West Ham. The fallout took months to clear up and led to a ban on investors acquiring the transfer rights of players in the English championship.

Infantino went looking for more information about the companies bankrolling the booming transfer market during the financial crisis. He told UEFA president Michel Platini, who said he was horrified to learn that so many footballers in Europe were being used as a type of financial product, often without their knowledge.

The former English footballer in Manchester, who now ran an investment fund, told us that clubs were merely raising money against their assets, like other businesses. The career of players was not affected, he said.

Infantino's enquiries gathered pace like a “snowball”, he later told us, growing as the size of the industry emerged. By 2013, investors owned stakes in the transfer rights of 1,100 players in Europe worth $1.5 billion, according to KPMG, a financial consultancy.

Over the next two years, Platini put pressure on FIFA president Sepp Blatter to ban the practice and on 1 May 2015, it was prohibited worldwide. This book tries to piece together the 30-year history of what we are calling football's secret trade, and what happened next.