Value from Data

Data and computer programs have a close relationship. A program often takes in one set of data—for example, a grid of temperature, humidity, and atmospheric measurements over a region of ocean—and transforms it into another set of data. Perhaps a storm track or intensity prediction. Absent data—and, in this case, data that’s not too old—the most powerful supercomputer and sophisticated weather model is worthless.

Data has long had value of course, but it was in the mid- to late 2000s that a widespread appreciation of that fact began to really sink in.

In 2006, Clive Humby, a UK mathematician and architect of Tesco’s Clubcard, coined the since widely quoted phrase: “Data is the new oil.” Less quoted is his follow-up, which emphasized that raw data has to be processed and analyzed to be useful. “It’s valuable, but if unrefined it cannot really be used. It has to be changed into gas, plastic, chemicals, etc to create a valuable entity that drives profitable activity; so must data be broken down, analyzed for it to have value,” he said.

However, in 2008, Wired magazine’s Chris Anderson wrote an article titled “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete.” His thesis was that we have historically relied on models in large part because we had no other choice. However, “with enough data, the numbers speak for themselves.”¹

It was a provocative point at the time and it remains a likely deliberate overstatement even today. However, the intervening decade has seen dramatic increases in machine learning performance across a large number of areas. For example, in certain types of tests, computers can actually outperform humans at image recognition today.

While the techniques to use data effectively continue to be refined, many of the most impressive “AI” achievements draw heavily from work that Geoff Hinton did back in the 1980s on a generalized back-propagation algorithm for training multilayer neural networks. These advances have come about partly through increases in compute power, especially in the use of graphics and specialized processors like Google’s Tensor Processing Units (TPU). They also depend on huge datasets used to train and validate the machine learning models.

It’s not as simple as saying “it’s all about the data” but there’s clearly been a shift in value toward data. As a result, companies like Google and Facebook are far more amenable to participating in open sourcing code than they are at opening their data (and the specific ways they work with that data such as search algorithms).

One example of open source principles and practices being applied to data is OpenStreetMap.

Steve Coast founded the project in 2004, which was initially focused on mapping the United Kingdom, where Ordnance Survey mapping data wasn’t freely available. (By contrast, in the United States, map services and data downloaded from The National Map maintained by the US Geological Survey (USGS) are free and in the public domain.) In April 2006, the OpenStreetMap Foundation was established to encourage the growth, development, and distribution of free geospatial data and provide geospatial data for anybody to use and share. OpenStreetMap is a particularly good example of open data because the data generated by the OpenStreetMap project is considered its primary output rather than map tiles or other services that can be created from the data.

According to the project, “OpenStreetMap emphasizes local knowledge. Contributors use aerial imagery, GPS devices, and low-tech field maps to verify that OSM is accurate and up to date.” It also allows for automated imports of data from sources that use appropriate licenses.

The resulting data is under the Open Database License. It is free to use for any purpose so long as OpenStreetMap and its contributors are credited.

Comparing the quality of OpenStreetMap with commercial services such as Google Maps is difficult because quality tends to be a function of both the location and which features are important to you. For example, OpenStreetMaps lacks the real-time information about road and traffic conditions needed to do effective routing. On the other hand, commercial services often draw data from sources that don’t emphasize features such as hiking trails which are more likely to be represented on OSM.

In general, though, it’s fair to say that both the quantity and quality of data in OpenStreetMap has improved dramatically over the past decade or so, and it’s a valuable open data resource for many purposes.

Transparency Through Data

Data can be valuable in the monetary sense. It enables services that are useful and for which consumers and organizations are willing to pay. As machine learning seems poised to be an integral part of more and more technology from automobiles to conversational assistants like Amazon’s Echo, data seems certain to only become more valuable—leading to increased tensions between opening data and profiting from it.

However, another aspect of data openness is the increased availability of data from public institutions and others so that citizens can gain insight into government actions and the environment they live in.

There are nuances to open data in this context.

Not all data collected by a government is or should be publicly available. The specifics will differ by country and culture—for example, salary information is more public in some countries than in others—but individuals and institutions will always (properly) have their secrets.

Furthermore, when data is opened, it needs to be done in a way that it’s actually useful. Red Hat’s Melanie Chernoff writes that “What we, as policy advocates, want to encourage is that the data that governments do and should publish is done so in a way to ensure equal public access by all citizens. In other words, you shouldn’t have to buy a particular vendor’s product in order to be able to open, use, or repurpose the data. You, as a taxpayer, have already paid for the collection of the data. You shouldn’t have to pay an additional fee to open it.”²

It’s easy to find fault with governmental and corporate transparency generally. However, the overall trajectory of making public data available is mostly positive.

For example, in May of 2013, then US president Obama signed an executive order that made open and machine-readable data the new default for government information. Over a quarter of a million datasets are currently available.³

Many municipalities also make available various types of data available about crime, traffic infractions, health code violations, zoning, and more.

How is this data used?

A common technique is to “mash up” data with maps. Humans are visual creatures, so presenting temperatures, crime statistics, or population densities on a map often makes quickly discerning patterns and spatial relationships easier than presenting the same facts as a boring table.

Some uses are mostly tactical and practical. USGS river data level is used by whitewater paddlers and others to plan their trips. National Weather Service, or specifically the forecasts based on it, help you decide whether to pack an umbrella in the morning.

However, other types of data can give insight into the “health” of cities or actions taken by public employees. One example from New York City revealed that police officers were routinely ticketing cars in locations where it was, in fact, legal to park under an amended law.⁴ Other examples include heat maps of where different types of crime occur. Individuals and companies can also build on public data to create applications that make it easier to use mass transit effectively or to more easily find out whether a restaurant has been cited for health concerns.

A common theme is that combining a multiplicity of datasets can often yield nonobvious insights.

Ownership of Data

The flip side to transparency and one that is much on the minds of regulators and others as of 2018 is the ownership of data and privacy issues associated with it.

Eben Moglen of the Software Freedom Law Center once referred to Google and its ilk as a “private surveillance system that [the government] can subpoena at will.” He went on to describe this as a “uniquely serious problem.” It’s hard to dispute that such services create an unprecedented centralization of data—both for data explicitly placed into their cloud and that generated with each search or purchase. This is anathema to those who saw open source and inexpensive computers as a great victory for the decentralization of computing and information.

The monetization of essentially private behaviors like browsing the Web for advertising and marketing purposes is part of the impetus behind regulations such as the European Union’s GDPR. The widespread availability and archiving of so much data have also exposed some of the fault lines between overall US and European attitudes toward free press, free speech, and privacy.

At a higher level, societies as a whole have also not really come to grips with what it means for so much data about individuals to be so widely available and combinable. It’s easy to focus on the usual suspects. Facebook. Equifax. Google. They’re profiting from mining our online behaviors and selling the results to advertisers and marketers. They make convenient bogeymen. (Perhaps some of their behaviors should even be curtailed.)

But there are broader questions. As we’ve seen, sharing data can be a positive. Speaking at the 2018 MIT Sloan CIO Symposium in May, Elisabeth Reynolds, the executive director of the Work of the Future Task Force, observed that “the regulatory framework is often opposed to sharing data for the greater good.”

For example, data about vehicle traffic, electricity usage, or personal health metrics could potentially be aggregated and studied to reduce congestion, optimize power generation, or better understand the effectiveness of medical treatments.

However, the more fine-grained and comprehensive the data set, the harder it is to truly anonymize. The difficulty of effective anonymization is well-known. Either your search history or your car’s GPS tracks would likely leave little doubt about who you are or where you live.

The problem is only compounded as you start mixing together data from different sources. Gaining insights from multiple data streams, including public data sets, is one of the promises of both IoT and AI. Yet, the same unexpected discoveries made possible by swirling enough data together also make it hard to truly mask identities.

Furthermore, there’s personal data that’s public for sound policy reasons, often to do with transparency. The sale of your house and your deed are public documents in the United States. But there’s a difference of degree that’s sufficient to become a difference of kind when all those public records are a mouse click away and ready to be combined with countless other datasets rather than filed away deep in some dusty county clerk’s office.

Openness of data is mostly a good thing. But it can be hard to strike the appropriate balance. Or even to agree what the appropriate balance should be.

The Read/Write Web

One aspect of this shift was “Web 2.0,” a term coined by O’Reilly Media to describe a web that was increasingly characterized by connections, which is to say a form of data describing relationships. In Tim O’Reilly’s words, “Web 2.0 doesn’t have a hard boundary, but rather, a gravitational core. You can visualize Web 2.0 as a set of principles and practices that tie together a veritable solar system of sites that demonstrate some or all of those principles, at a varying distance from that core.”⁵

Another way to look at this phase comes from the inventor of the World Wide Web, Tim Berners-Lee, who used the term “read/write” web to describe a web in which everyone created as well as consumed.⁶ This was around the time that blogging was really taking off and content created by users became an important part of the data and information underpinning the web.

Besides the Web as a whole, the best example of a large-scale collaboratively created open information resource is probably Wikipedia.

Wikipedia

Launched in 2001 by Jimmy Wales and Larry Sanger, Wikipedia wasn’t the first (or last) attempt to create a crowdsourced encyclopedia. In a 2011 article, Megan Garber, writing for NiemanLab, counted six prior attempts. Subsequent attempts at different takes on the Wikipedia model, such as Google’s Knol, were likewise unsuccessful.

The reasons for Wikipedia’s success have never been completely clear. In 2011, research by Berkman fellow and MIT Media Lab/Sloan School of Management researcher Benjamin Mako Hill⁷ suggested it attracted contributors because it was a familiar product (an encyclopedia), focused on content rather than technology, offered low transaction costs to participation, and de-emphasized individual ownership of content.

One also suspects that it stumbled into a governance model that struck at least a workable balance between autocratic control structures and a crowdsourced free for all. There’s no shortage of complaints, even today, about various aspects of how Wikipedia is administered and certainly quality can be uneven. However, it’s hard to dispute the fact that Wikipedia mostly works and is a valuable resource.

Independent Contribution

Other sites are open and collaborative in the sense that they provide a platform for many users to contribute content, under a Creative Commons open source license or otherwise. YouTube is well-known for videos. Flickr, recently purchased by SmugMug, is one example for photos although—in terms of sheer numbers—more social media-oriented properties like Instagram (owned by Facebook) are much larger.

The differences in the collaborative approach between Wikipedia and these other sites are probably rooted in the type of content. Collaboratively edited encyclopedia articles may suffer a bit from not having the coherence that a single author or editor can bring. But diverse perspectives that also guard against bias and self-promotion are usually a net win. On the other hand, collaboration on a posted photograph or video doesn’t really make sense in most cases other than remixing or incorporating into new works.

Another large digital library is The Internet Archive, founded by Brewster Kahle in 1996, which has a stated mission of “universal access to all knowledge,” including nearly three million public domain books. The Internet Archive allows the public to upload and download digital material, but the bulk of its data is collected automatically by its web crawlers, which work to preserve copies of the public web except where owners explicitly request otherwise. This service, which allows archives of the Web to be searched and accessed, is called The Wayback Machine, a play on the time machine in “Peabody’s Improbable History,” a recurring feature of the 1960s cartoon series The Rocky and Bullwinkle Show.

Certainly, vast quantities of information remain behind paywalls and firewalls—or locked up on paper, microfilm, or microfiche—but the amount of information that is readily available with limited restrictions on its use would still be almost unfathomable just a few decades ago.

Precursors

In the United States, an early example of what starts to look like an open education program comes from 4-H clubs. (4-H derives from the organization’s original motto: head, heart, hands, and health.) It grew out of a desire to expose children to practical and “hands-on” learning that connected public school education to country life—and, in turn, expose their parents to the new agricultural approaches and technologies they tended to resist. According to the organization, “A. B. Graham started a youth program in Clark County, Ohio, in 1902, which is considered the birth of 4-H in the United States. The first club was called ‘The Tomato Club’ or the ‘Corn Growing Club.’ T.A. Erickson of Douglas County, Minnesota, started local agricultural after-school clubs and fairs that same year. Jessie Field Shambaugh developed the clover pin with an H on each leaf in 1910, and by 1912 they were called 4-H clubs.”

4-H and related programs persist in many rural and semi-rural communities today as seen in the many country fairs that are a summer and autumn staple in parts of the United States such as the town where I live.

MIT OpenCourseWare

Fast forward to the Internet age and hit pause at 2001. That was the year that the Massachusetts Institute of Technology first announced MIT OpenCourseWare, which is an obvious point at which to start our look at where open education stands today. It was formally launched about two years later.

The concept grew out of the MIT Council on Education Technology, which was charged by MIT provost Robert Brown in 1999 with determining how MIT should position itself in an environment where other schools were increasingly offering distance learning options to paying students.

MIT took a rather different approach. For one thing, the content was made available for free under an open license.

As Carey Goldberg of The New York Times reported, Steven Lerman, the faculty chairman, argued that “Selling content for profit, or trying in some ways to commercialize one of the core intellectual activities of the university seemed less attractive to people at a deep level than finding ways to disseminate it as broadly as possible.”⁸

For another, the emphasis was on providing the raw ingredients for educators rather than an education in and of itself. In announcing the initiative, MIT President Charles Vest said that “We are not providing an MIT education on the Web. We are providing our core materials that are the infrastructure that undergirds an MIT education. Real education requires interaction, the interaction that is part of American teaching. We think that OpenCourseWare will make it possible for faculty here and elsewhere to concentrate even more on the actual process of teaching, on the interactions between faculty and students that are the real core of learning.”

Many other schools announced programs in the same vein over the next few years and, in general, these commons of educational resources have grown over time.

We’ll return to this idea of open educational resources (OER) shortly but, first, no discussion of open education would be complete without mentioning massive open online courses (MOOCs).

MOOCs

As Vest noted, the commons populated by MIT OpenCourseWare and other resources of its type were intended as resources for learning rather than a course in a box. In addition to the reasons Vest gave, one suspects that the additional distance this approach provided between in-person MIT courses and MIT OpenCourseWare ones made the concept an easier sell. At a practical level, ubiquitous video and audio recording of lectures and other events also weren’t yet a thing.

However, this wasn’t what a lot of potential students were looking for; they wanted a virtual version of a university course. As the 2000s progressed, many were becoming accustomed to watching instructional videos and lectures on YouTube. Various academics and others were also starting to see an opportunity to broaden the reach of elite university educations to populations that were underserved in their access to that level of education.

Precursors had been around for a while, but it was 2012 when MOOCs exploded including both venture capital-funded (Coursera and Udacity, both with Stanford professors attached) and non-profit (edX, initially begun by MIT and Harvard). As The New York Times’ Laura Pappano wrote at the time, “The shimmery hope is that free courses can bring the best education in the world to the most remote corners of the planet, help people in their careers, and expand intellectual and personal networks.”

Courses were free and sign-ups were often massive—up to 150,000 or so massive—although the drop-out rate was correspondingly high, with numbers in excess of 95 percent common. Not really surprising given the lack of any real commitment required to hit the register button.

MOOCs are still around but their record has been mixed.

From an open source perspective, they always seemed closer to free as in beer, rather than free as in speech. By design, most MOOCs follow a fairly traditional format for a university course with short talking-head and virtual whiteboard videos standing in for the lecture hall experience. Grades (remember extrinsic motivation) mostly come from various autograded tests and homework including multiple choice, numerical answers, and computer program output.

Courses often do encourage participation in discussion forums and they often enlist past students as teaching assistants. However, many courses don’t follow a rigid schedule. Furthermore, the practical realities of holding meaningful discussions among thousands of students at vastly different levels of educational background and language ability make MOOCs much more of a broadcast medium than a participatory one, much less one where participants can choose what material is covered, learn from each other, or even guide the overall course direction.

For the VC-funded Udacity and Coursera in particular, the difficulties of making money through giving content away for free also became troublesome. A number of MOOCs started out by offering a “verified certificate”—using the same sort of tools used for online proctoring of other types of certifications—for a fee. The problem was that a lot of employers didn’t see a lot of value in a verified certificate relative to an unverified one, assuming they saw value in completing a MOOC at all. Over time, MOOCs have generally come to make all grades and certificates paid add-ons; some have even eliminated tests and other exercises from the free version. In 2013, Udacity would “pivot” (in Silicon Valley jargon) to simply charging for classes and focusing on vocational training.

This last point reflects what much of the audience for MOOCs ended up becoming anyway. For example, in 2013, after six months of high-profile experimentation, San Jose State University “paused” its work with Udacity because students in the program actually were doing worse than those in normal classes.⁹ The student body of a typical MOOC tends to be populated heavily with early- to mid-career professionals, often with advanced degrees. On the one hand, MOOCs continue to be a great resource for those educated and self-motivated learners. But they’ve been a bitter disappointment for those who saw MOOCs as a remedy for the barriers high prices and inefficiencies of traditional university education erected against those who lack money and family support.

Collaboration versus Consumption

There are those who argue that MOOCs were a bad model anyway.

In 2013, Rolin Moe wrote that: “Had Udacity been able to provide a modicum of quality education to underrepresented populations, criticism would have remained to argue the pedagogy of such learning. With Udacity shifting away from underrepresented populations, the criticism is now about what is left in the wake of 2+ years of hyperbole and 0 years of results. And we cannot confuse what has shifted. It is not the narrative but only their business model; the narrative is still a system in crisis and searching for some low-cost, tech-savvy gizmo to do the job because we only need 10 schools and unions are the problem and our kids don’t know enough STEM and plumbers make more money than college grads anyway.”¹⁰

MOOCs essentially solved the problem of bringing a lecture to those who can’t be physically present. But that’s been a more or less solved problem since the advent of VHS tape. MOOCs make the lecture more consumable but it’s still basically just a lecture.

Martin Weller has written about how connectivism as a learning theory “as proposed by George Siemens and Stephen Downes in 2004–2005, could lay claim to being the first internet-native learning theory. Siemens defined connectivism as ‘the integration of principles explored by chaos, network, and complexity and self-organization theories. Learning is a process that occurs within nebulous environments of shifting core elements—not entirely under the control of the individual.’” He went on to say that “What was significant about connectivism was that it represented an attempt to rethink how learning is best realized given the new realities of a digital, networked, open environment, as opposed to forcing technology into the service of existing practices.” Yet, “while connectivism provided the basis for MOOCs, the approach they eventually adopted was far removed from this and fairly conservative.”¹¹

My Red Hat colleague Gina Likins argues that we should be thinking about education in terms of broader participation. For example, “people should be expected to fork the educational materials. What one classroom needs can’t be expected to work in another. Students will come from different backgrounds with different experiences. The textbook is only the start.”

Likins also points to other examples of open source community development that could apply to education. For example, many educational materials—textbooks in particular—follow a philosophy of not releasing before it’s final. There are valid reasons for some of this. And, especially at the secondary school level, there are complex political considerations with some topics like history as well. But it’s another way in which educational resources are developed with feedback from only a relatively small insular group, often with a limited set of perspectives.

Academic research is also seeing open access movements. Some institutions have adopted open access policies to grant the public access to research materials. The Public Knowledge Project maintains an open source publishing platform called Open Journal Systems, which editorial teams can use to referee and publish (largely open access) academic journals outside the traditional publishing system.¹² It’s also become more common to publicly circulate drafts and other pre-print versions of research as a way of soliciting broader feedback.

Ham Radio

One early example of open source-ish sharing of hardware designs is amateur radio. The common term “ham radio,” as amateur radio came to be known, was actually born of a slur. Professional wired telegraph operators used it in the 19th century to mock operators with poor Morse code sending skills (“ham-fisted”) and it carried over to the amateurs experimenting with wireless telegraphy at about the beginning of the 20th century.

Factory-built gear wasn’t readily available as ham radio was getting started, so amateurs began handcrafting vacuum tube-based transmitters and receivers. After World War II, surplus military gear also became widely available.

Amateur radio publications encouraged this sort of grassroots experimentation with hardware. In Ham Radio’s Technical Culture (Inside Technology) (MIT Press, 2008), Kristen Haring recounts how, in 1950, CQ Amateur Radio Magazine announced a “$1000 Cash Prize ‘Home Brew’ Contest” and called independently-built equipment “the type of gear which has helped to make amateur radio our greatest reservoir of technical proficiency.”

This hobbyist homebrew culture gave rise to the once widespread Radio Shack chain. Started in in 1921 by two brothers, Theodore and Milton Deutschmann, it provided equipment for the then-nascent field of ham radio from a downtown Boston retail and mail-order operation. The “radio shack” term came from a small, wooden structure that housed a ship’s radio equipment.

The company had ups and downs and ownership changes over the years before its owners largely shut it down in 2017. However, for a few decades after its acquisition by Tandy in 1963, RadioShack stores (as they were called by then) were popping up in shopping malls and city centers across the United States and elsewhere. And, in so doing, becoming a sort of home for electronics enthusiasts of all stripes, including early personal computer hobbyists. The TRS-80, introduced in 1977, was one of the first mass-produced PCs and initially outsold the Apple II by harnessing the power of RadioShack’s retail channel and its thousands of locations.

To be sure, some of the nostalgia is selective. The company may have been the most convenient place for what we now call “makers” to pick up a needed resistor or capacitor. But its ubiquity also made it the default for often less-knowledgeable consumers to pick up subpar audio products and other consumer electronics in the days when the alternative was usually either a specialty retailer or a department store.

A Shift from Making

RadioShack was doomed in large part by some of the broad changes in electronics retailing dating to roughly the beginning of the 21st century. E-commerce, big box stores, and the emergence of smartphones were all in that mix and RadioShack never really adapted.

However, there were also technology shifts happening that affected the DIY tinkerers who viewed RadioShack so fondly even while the company was still successful in the 1990s.

Through about the 1980s or so, it was possible to design, build, and share plans for nontrivial electronics projects more or less from scratch. The Heath Company sold electronic test equipment like oscilloscopes, home audio gear, TVs, amateur radio equipment, and more in kit form under the Heathkit brand. A certain satisfaction and knowledge (if often frustration!) came from soldering and hand-assembling a complete working device from components, even if you were just working from cookbook instructions.

Byte, a relatively mainstream computer magazine, carried a regular column by Steve Ciarcia called “Circuit Cellar” that introduced a variety of electronics projects that readers could build. As in the case of the code the computer magazines printed, there was usually no explicit license attached to these types of designs, but there was an implicit understanding that they were there to use, modify, and share.

The first thing to affect the traditional electronics DIYers was the personal computer. Many early PCs were something of a DIY project on their own. But it was more in the vein of assembling boards and other prebuilt parts such as disk drives and then figuring out why the thing wouldn’t work. The process involved problem solving and learning, but it was certainly different from creating from scratch.

But PCs, after you got one working, also drew many hobbyists from hardware to software. To many, software seemed to provide more opportunities to build “real” things. It was certainly easier to share your work with others who could benefit from it. Although open source software in a formal sense wasn’t very widely known in the 1980s, “freeware,” “shareware,” and source code published for educational purposes widely circulated on bulletin board systems and on disks sold at computer shows.

It was also the case that electronics were simply getting harder to tinker with. Components were miniaturizing. They were getting faster and more complex. The devices that most people were in a position to design and build at home looked increasingly primitive compared to what you could buy off-the-shelf, often for less money.

The New Makers

Things remained in more or less this state until the mid-2000s.

The Arduino project started in 2003 as a program for students at the Interaction Design Institute Ivrea in Ivrea, Italy. Its goal was to provide a low-cost and easy way for novices and professionals to create devices that interact with their environment using sensors and actuators. The project’s products are distributed as open source hardware and software, allowing anyone to manufacture Arduino boards and distribute the software.

In addition, being open source, Arduino was significant because it offered a good model for hobbyists and other to practically build interesting hardware projects in the modern era. Arduino board designs use a variety of microprocessors and controllers and are equipped with digital and analog input/output (I/O) pins that may be interfaced to various expansion boards and other circuits (Figure 7-1). Effectively, an Arduino embeds and abstracts away a lot of the complexities involved with interfacing with the physical world.

One person who recognized this was Limor Fried, who started Adafruit in her MIT dorm room during 2005 to sell electronics components and kits, including Arduinos. Now, it’s a New York City company that she runs to make open source electronics kits and components for the growing tide of DIYers. For example, you can build a robot or a giant timer display.

In a 2011 interview with Wired magazine, Fried said that “One of the things about doing projects is that documenting them and sharing them with people used to be really difficult. Now we see so many people putting up videos, and it’s so easy. I can make a five-minute video in an hour, with a preface and edits and nice audio and everything. And I think that makes a difference, because when people are exposed to this stuff and they actually get to see it, they get inspired.”

Another more recent hardware project is the Raspberry Pi, which is, in effect, a miniaturized low-cost Linux computer that can also interface with external components. First released in 2012, this series of small single-board computers was developed in the United Kingdom by the Raspberry Pi Foundation to promote the teaching of basic computer science in schools and in developing countries. Unlike Arduino, the Raspberry Pi computer is not itself open source, but much of the software that it runs, including the operating system, is as are the plans for many projects that make use of the Raspberry Pi.

At about the same time as these computers for makers were becoming popular, buzz was also building for 3D printing. 3D printing is a subset of additive manufacturing in which material is selectively deposited based on the instructions in a 3D design file. Imagine thin layers of (typically) plastic printed one at a time until an entire three-dimensional object comes into existence.

3D printing is usually dated to Chuck Hull’s 1986 invention of a stereolithography apparatus (SLA). However, from an open source and maker angle, the most relevant date is 2007, when Adrian Bowyer of the University of Bath founded the RepRap project, an open source project intended to build a 3D printer that could print most of its own components.

Over the next few years, commercial 3D printing started to roll out. One early company, MakerBot, pulled back somewhat from its open source roots—although more recently it’s rolled out a Labs platform as a way to deal with criticism for its closed source stance.¹³

3D printing has proven popular as a way to economically create one-off parts. This is handy for a hobbyist to create a one-off case, gear, or just something decorative. However, in addition to its use in certain types of manufacturing, 3D printing is also used to create custom prosthetics and other medical devices.

Arguably, 3D printing hasn’t yet lived up to the hype that’s sometimes surrounded it, but it provides another example of how hardware designs can be cooperatively developed and shared.

Why Do Organizations Exist Anyway?

“The Nature of the Firm” is a seminal 1937 article¹⁴ by Ronald Coase, a young British economist who would go on to win the Nobel Prize in economics over the course of a long career. Coase asked the questions: Why do people give up some of their freedom to work as they like and become full-time employees? And wouldn’t it be more efficient for firms to depend on loose aggregates of private contractors, hired as-needed to perform a specific task? After all, orthodox economics going back to at least Adam Smith suggested that everyone should already be providing goods and services at the best rate available as part of an efficient market. Why hire someone who you might need to pay even at times when you don’t have an immediate need for their skills and services?

His answer was that it’s to avoid transaction costs associated with using a market-based price mechanism.

You need to find a person who can do something you want done but that you can’t or don’t have time to do on your own. You need to bargain with them. You need to trust that they will keep your trade secrets. You need to educate them about your specific needs. So, firms get created, and often grow.

Of course, it’s a balancing act. Firms have always used specialist suppliers—think ad agencies for example—which can be thought of as reducing transaction costs themselves relative to contracting with individuals off the street. As The Economist noted on the occasion of Coase’s 100th birthday in 2010, however: “Mr Coase also pointed out that these little planned societies impose transaction costs of their own, which tend to rise as they grow bigger. The proper balance between hierarchies and markets is constantly recalibrated by the forces of competition: entrepreneurs may choose to lower transaction costs by forming firms but giant firms eventually become sluggish and uncompetitive.”¹⁵

These ideas were built on by many economists over the years, including Oliver E. Williamson who once observed that it is much easier to say that organizations matter than it is to show why or how; he would also win a Nobel Prize for "his analysis of economic governance, especially the boundaries of the firm," which he shared with Elinor Ostrom.

More recently, New York University Professor of Law Yochai Benkler explicitly argued in a 2002 paper that “we are beginning to see the emergence of a new, third mode of production, in the digitally networked environment, a mode I call commons-based peer production.” This third mode, the successor to firms in companies and individuals in markets, arose from open source software, he said.¹⁶

It’s at least interesting to ask how online markets, connected communities, and the “gig economy” (think Uber) change historical equations. We’ve seen the rise of coopetition earlier in this book. There’s little doubt that contracting out for certain types of work has indeed become easier for companies—for better or worse. On the other hand, while public clouds have become a generally beneficial option for certain types of computing workloads, there are also plenty of outsourcing horror stories in IT. We should perhaps leave this question at “it depends” and refer back to examples of how open source communities best work as the relevant discussion points for this book.

However, it’s also worth considering what open source practices and principles mean within a given firm. Red Hat CEO Jim Whitehurst refers to this as the “open organization” in a book of the same name (The Open Organization: Igniting Passion and Performance, Harvard Business Review Press, 2015).

Open Organizations

General Motors might seem an odd starting point for this discussion, but, for all its various problems over the years, it provides an interesting study point about the principles of decentralization, which is a component of openness.

Writing in 2018, Steve Blank noted how “Borrowing from organizational experiments pioneered at DuPont (run by his board chair), Sloan organized the company by division rather than function and transferred responsibility down from corporate into each of the operating divisions (Chevrolet, Pontiac, Oldsmobile, Buick and Cadillac). Each of these GM divisions focused on its own day-to-day operations with each division general manager responsible for the division’s profit and loss. Sloan kept the corporate staff small and focused on policymaking, corporate finance, and planning. Sloan had each of the divisions start systematic strategic planning. Today, we take for granted divisionalization as a form of corporate organization, but in 1920, other than DuPont, almost every large corporation was organized by function.”¹⁷

GM was still the epitome of a hierarchical organization, of course, in the vein of the companies that William Whyte wrote about in 1956 in his influential The Organization Man (Simon & Schuster). A central tenet of the book is that average Americans subscribed to a collectivist ethic rather than to the prevailing notion of rugged individualism. The Fortune magazine writer argued that people became convinced that organizations and groups could make better decisions than individuals, and thus serving an organization was a better logical choice than focusing on individual creativity.

GM was a hierarchical organization distributed through accounting structures rather than direct command and control.

Today, we see some signs of broader change.

More democratic forms of organizational governance exist. One popular discussion topic is holacracy, which introduces the idea of roles (rather than job descriptions), as part of a system of self-organizing, although not self-directed, circles. The term was coined by Arthur Koestler in his 1967 book The Ghost in the Machine (UK: Hutchinson; US: Macmillan). It’s now a registered trademark of HolacracyOne although the model itself is under a Creative Commons license. The best-known practitioner is probably Zappos, the online shoe retailer now owned by Amazon. In a 2015 memo, CEO Tony Hsieh wrote “Holacracy just happens to be our current system in place to help facilitate our move to self-organization, and is one of many tools we plan to experiment with and evolve with in the future. Our main objective is not just to do Holacracy well, but to make Zappos a fully self-organized, self-managed organization by combining a variety of different tools and processes.”

However, the prevailing wisdom—which reflects practices familiar to open source development practitioners—is not so much about democracy as it is about decentralization and empowerment based on skills and expertise.

In The Open Organization, Red Hat’s Whitehurst argues that “Many people assume that if an organization is not top-down that it must be some flavor of democracy—a place where everyone gets a vote. In both hierarchies and democracies, decision-making is clear and precise. it’s proscribed and can be easily codified. In most participative organizations, however, leaders and decision-making don’t necessarily follow such clear rules, just as it was in ancient Athens, where literally every citizen had the same opportunities to lead or follow others. Some people have more influence than others.”

One interesting aspect of such an environment is that it doesn’t necessarily mean, as one might assume it would, eliminating managers. Whitehurst writes that “Nothing could be further from the truth. Our managers play a vital role in building, supporting, and moderating the meritocracy. Finding that balance between supporting it and, at the same time, leaving things alone is critical.” Think back to our discussion of the role of maintainers like Greg Kroah-Hartman in the Linux kernel and this dynamic will seem familiar.

The need to delegate and federate decision making isn’t a new insight. It was pushed down to the divisional level at GM under Sloan. Management consultant Gary Hamel argues, “building and benefiting from communities at scale requires us to start with ‘openness’ as a principle,’ rather than with some particular set of collaborative tools or practices.” It’s more pervasive than an organizational delegating decision making within a formal structure.

In Team of Teams: New Rules of Engagement for a Complex World (Portfolio, 2015), Stanley McChrystal, who commanded the US Joint Special Operations Command in the mid-2000s, describes the basic problem. He recounts how the scientific management system that Frederick Taylor unveiled at the 1900 Paris Exposition Universelle “was so beautiful it inspired people to devote their lives to his vision.” It was so impressive for how efficient it was at executing known, repeatable processes at scale. His steel model could churn out metal chips at a rate of 50 feet per our rather than the norm of nine.

However, McChrystal goes on to write, today’s world is more interdependent. It moves faster.

This creates a state of complexity, which is fundamentally different from challenges that are “merely” complicated in a technical sense. Complexity in this sense means less predictable. It means emergent behaviors that increasingly need to be reacted to rather than planned for in great detail.

McChrystal argues for pivoting away from seeing efficiency as the managerial holy grail to a focus on adaptability. He distinguishes commands rooted in reductionist perfection from teams that have a connectivity of trust and purpose that gives them an ability to solve problems that could never be foreseen by a single manager—even if they’re less efficient in some theoretical sense.

All of which echoes the open source development model in so many ways.