Jolyon Holdstock

10.1 Introduction

Founded in 1995 by the pioneer of Southern blotting and microarray technologies, Professor Sir Edwin Southern, Oxford Gene Technology (OGT) is focused on providing innovative clinical genetics and diagnostic solutions to advance molecular medicine. OGT is based in Yarnton, Oxfordshire and is a rapidly growing company with state-of-the-art facilities and over 60 employees. The key focus areas of OGT are:

Here, I will describe the approach that OGT has taken in the development of our in-house, ’omics platform. This covers the initial stages of setting up a commercial platform to service customer microarrays, through to developing our own microarray products, running high-throughput projects to recently setting up a pipeline to assemble and annotate data from next-generation sequencers. Each of these has required a range of software applications. Whenever OGT requires new software, a standard approach of evaluating all possible options is followed, be they FLOSS, commercial or even commissioned; weighing up the advantages and disadvantages of each solution. Obviously an important consideration is cost, and where there are comparable products then this becomes an important part of any discussion to decide which products to use. However, cost is just one of a series of important factors such as ease of use, quality of output and the presence/absence of specific skills within the group. Furthermore, any likely impact on infrastructure must also be borne in mind before a decision is made. In reality, more often than not a FLOSS alternative will be selected, indicating just how much high-quality FLOSS software is now available.

10.2 General changes over time

The computational biology group at OGT has evolved hugely over this time. In 2003, OGT was a small biotechnology company with a staff numbering in the low 20s, servicing mostly small academic projects for customers within the UK. Since then, the company has grown to more than 70 full-time employees and has customers from both industry and academia based around the globe. Concomitantly, the typical value of projects has greatly increased from the low thousands of pounds involving a handful of arrays, to hundreds of thousands based on thousands of microarrays. To cope with this growth, our internal procedures and processes have had to develop and evolve.

Indeed, the consequences of the improvements in microarray technology have been the increased demands placed on the IT infrastructure that supports it. For example, in 2003 the densest microarray that OGT produced contained 22 500 oligonucleotide probes. Following hybridisation and processing, this generated a data file of about 12 MB. Since then, DNA spotting has improved such that more features can be printed, and the resolution of scanners has similarly increased so these features can be smaller. This frees up real estate on the slide allowing for increases in feature number. At the time of writing, the latest arrays from our microarray provider, Agilent, have 1 000 000 features and generate a data file around 1.4 GB. This dramatic increase impacts across the IT system; files take longer to move around the network, backups take longer to run and files are now too big to deliver by email or via traditional methods such as FTP. Furthermore, we have recently expanded our offerings to include next-generation sequencing (NGS). This has necessitated a large investment in hardware to provide the processing power to deal with the terabytes of data generated. The chapters by Thornber (Chapter 22) and MacLean and Burrell (Chapter 11) provide good background on NGS data for those unfamiliar with this topic.

10.3 The hardware solution

Within the company there is a requirement for our hardware system to integrate the needs of the computational biology team, who use both Windows and Linux, with those outside the group, who work purely within a Windows environment. For example, the PCs in the labs used to process the microarrays must be able to write to a location that can be accessed by the Linux-based applications used to process the data. The approach taken at the start of OGT, and still in place today, was to keep all data on disk arrays controlled by a Linux server. These directly attached storage (DAS) arrays mounted on the Linux platform are then made available to all users over the company network. This is accomplished using the Linux utility Samba [2]. Samba comes with a great deal of supporting information to allow users to understand this powerful system. Essentially, there are folders defined within Samba to which Windows users can map a drive. The appearance of these drives is of a normal Windows folder, but under the bonnet the information is being made available across the network from a Linux server. This integration of Windows and Linux is reinforced by the fact that all authentications are handled by Windows Active Directory. It is the Active Directory that controls all aspects of user permissions, irrespective of the platform being used, making it much easier to manage the system. The only exceptions to this are root access to the servers, which is independent and will authenticate as usual via the standard/etc/ passwd mechanism.

Originally, the first hardware purchased for the OGT ’omics platform was a complete system comprising a Beowulf compute cluster of a master server and four nodes. The master server handled all processes and farmed out jobs to the nodes on the cluster. All data was stored on a 1 TB DAS disk array and there was tape library to handle backups. This is shown in Figure 10.1.

The advantage of buying a pre-configured system was that it was ready to go on arrival. It reduced the time required to get up and running and make the ’omics platform operational. Also, although some of the group had advanced Linux skills, these were short of the level needed to act as a true system administrator and setup a system from scratch. Following installation by our supplier, all day-to-day maintenance could be handled by a nominated member of the computational team who had access to an external consultant if required. The disadvantage of buying a pre-configured system became apparent when there was a problem after the warranty had expired. The master server developed a problem in passing jobs to the compute nodes. The supplier would no longer support the system and our Linux consultants could not resolve the issue because they did not understand how the cluster was meant to be working.

In the end, the situation was resolved by the purchase of a new server. With this server, and all subsequent servers, we have taken the approach of buying them without any operating system (OS) installed. We are therefore able to load the OS of our choice and configure the server to our own individual requirements. This can delay the deployment but it does mean when there are problems, and there have been some, we are much better able to resolve the issue.

Lesson: Understand your own hardware or know somebody who does.

This highlighted two other important points. First, the design of our system as it was with a single server meant there was no built-in redundancy. When the server went down, the disk arrays went down. Consequently, none of the Windows folders mapped to the disk array were available; this had an impact across the whole of the company. Second, it showed that IT hardware needs replacing in a timely manner even if it is working without fault. This can be a difficult sell to management as IT hardware is often viewed with the mantra ‘if it ain’t broke don’t fix it’; however, at OGT we take the approach that three years is the normal IT lifespan and, although some hardware is not replaced at this milestone, we ensure that all the hardware for our business critical systems is. This coupled with a warranty that provides next business day support, means that OGT has a suitable level of resource available in the event of there being any IT hardware issues.

Lesson: Take a modular approach and split functions between systems.

Subsequently, we have developed a more component-based approach of dividing operations between different servers so that even when one server goes down unexpectedly, other services are not affected. For example, there is a single server providing file access, another for running a BLAT server, another running our MySQL database, etc. This is illustrated in Figure 10.2.

10.4 Maintenance of the system

Responsibility for the day-to-day administration of the ’omics platform resides within the computational biology group. There is a nominated system administrator, but we also have a support contract with a specialist Linux consultancy who are able to provide in-depth knowledge when required. For example, when we buy a new server, they install the operating system and configure it for our needs. Also, they provide a monitoring service so that all our systems are constantly being assessed, and in the event of a problem being detected are alerted and able to resolve it. Further to this, they have been able to provide sensible and practical advice as we have expanded. From the OGT administrator’s point of view, this reduces time spent as a system administrator, freeing up time for other important functions in the group.

One early issue was that the Linux and Windows systems were maintained by separate people, with little communication flow in either direction. Given the overlap between these systems, this was not ideal. A recent development has been the formation of an IT steering committee comprising the Linux and Windows system administrators and some other IT literati from within OGT. With this has come regular fortnightly meetings at which any problems can be discussed, as well as any plans for the future discussed. Usually these meetings do not last very long, but they have significantly oiled the IT machinery within OGT.

Finally, with regard to being a system administrator, the following are some general rules of thumb I have found that hold true in our experience.

10.5 Backups

Given our commercial projects as well as our internal development projects, there is an absolute requirement to ensure that all necessary data are secured against the possibility of data loss. We settled on tapes as the medium for this, and our chosen backup strategy is to run a full backup of all data over the weekend and incremental backups Monday to Thursday. This covers all OGT’s current projects; once a project is completed it is archived to a separate tape volume and retained, but not included further in the weekly backups as it is no longer changing.

Initially, our pre-configured IT system was delivered with a commercial product installed, this was NetVault from BakBone [4]. As this was already installed and configured on the system, worked well and was easy to use, there was no need to change to an alternative. However, as the amount of our IT hardware expanded so did the licence requirements for running NetVault. As a result, we considered open source alternatives. The main candidates were Bacula [5] and Amanda [6] (the Advanced Maryland Automatic Disk Archiver). In the end, we chose to use Bacula primarily because it was the package with which our support was most familiar. Tools such as Bacula illustrate a common factor when considering FLOSS against commercial solutions, namely the level of GUI usability. However, in this case this was not such a factor as the console-based command-line controller offers a rich set of functionality. Although the learning curve is clearly steeper, this option means that administrators have no choice but to develop a better understanding of the tools capabilities as well as providing much finer control of the processes involved. Finally, the data format used by NetVault is proprietary and requires the NetVault software to access any previous archives. In contrast with this, Bacula provides a suite of tools that allow access to any archives without the requirement that Bacula is installed.

10.6 Keeping up-to-date

Taking this modular approach has helped significantly in managing the setup, as well as making the system much more robust and resistant to failure. Although the result is a more stable hardware platform, it has still been important to stay up-to-date with new technologies. For example, there is a great deal written about cloud computing and its impact on anything and everything (see Chapter 22 by Thornber, for example). Indeed, at a recent conference on ‘end-to-end’ infrastructure, most suppliers were offering products using virtualisation and cloud capabilities. This is something often discussed internally and something we feel does not offer any benefits significant enough to make it worth implementing at OGT at the moment. Of course, as this technology matures its capabilities will only increase and it will almost certainly have a future role to play at OGT. However, given the amount of data processed on a daily basis, the use of the cloud as a data repository is neither practical nor cost-effective at the moment. Where it may play a role for OGT is in providing a platform as a service; for example, if there was a short-term requirement for additional processing power that could not be met by our current systems. The ability of the cloud to provide such additional resources quickly and easily could prove very useful.

On a simpler level, there are alternatives to the DAS disk arrays currently in use. One of these is a network attached storage (NAS) device, as shown in Figure 10.3. As described previously, DAS disk arrays are used to provide the necessary data storage facility within OGT. However, some aspects of the system are being updated to include NAS devices as well. As the name suggests, a NAS device is a computer connected to the network providing storage, and, as such, performs a job similar to a DAS; but there are some significant differences. The key difference is that a NAS is a self-contained solution for serving files over a network, whereas a DAS is a set of disks made available by an existing server. The NAS units currently used already come with an embedded Linux operating system. Consequently, from the initial setup on the network through to its day-to-day management the NAS will require less administration than a DAS. Furthermore, all administration will be handled through GUI-driven management software. As such, it is close to a plug-and-play solution with all the necessary software pre-installed, for instance, for it to act as a file server, web server, FTP server or printer server. There is also the option of replication between NAS devices, which will further strengthen the platform. We employ one NAS box as the primary file server and use the replication software on the NAS to maintain a duplicate instance at another site. There are some disadvantages of using a NAS. As a pre-configured system it is generally not as customisable in terms of hardware software as a general-purpose server supplied with DAS. But, as all we require is a means to make files available over the network, the NAS is currently the best option.

10.7 Disaster recovery

One of the best definitions of disaster recovery is from Wikipedia [7] and is as follows:

‘What happens if the building is hit by a plane?’ is an often-quoted but fortunately rare scenario, but it remains the benchmark against which all recovery plans are measured. A perfect plan for disaster contingency will deliver zero data loss coupled with zero system downtime. Although this is always possible, such a level of protection will likely come at too high a price to be viable from a business perspective. At OGT, we have taken a flexible view of our needs and ensured that business critical systems have mechanisms in place to ensure they can be maintained. Other less important systems are given a lower level of support corresponding to their importance to the organisation.

There are different types of measure that can be employed to defend against disaster:

These controls should be always documented and tested regularly. Documentation is particularly important; if the recovery plan does not accurately describe the process to restoring the system, the value of the entire plan is significantly lessened. Modularity again plays a key role here, allowing the plan to be tailored for different elements of the system, such as those described below.

10.8 Personnel skill sets

The need for bioinformatics programming skills is becoming a necessity within biology and medicine [41]; there is now a requirement for scientists to have access to bioinformatics support and there is currently a lack of such expertise. As reported at the UK parliament by the Science and Technology Committee in their Second Report on Genomic Medicine [42], Professor Sir John Bell stated that ‘the really crucial thing to train in the UK will be bioinformaticians—people who can handle data—the truth is we have now hit the wall in terms of data handling and management’.

This requirement is due to the exponential growth in data being generated, coupled with the increasingly detailed questions being asked of it. Although there are some degree and postgraduate courses in bioinformatics, there is also the opportunity for personal development through self-education and experience. What was once defined as ‘would be nice to have’, bioinformatics programming skills are now more of an ‘essential’ across many fields of biology. This is due to an increased awareness of what may be done and also the massive increase in the amount of data being generated. This all needs to be processed, stored and re-used as required. Similarly, biological information is distributed now thus adding web skills to an already bulging list of requirements in the bioinformatics tool bag.

There are a wide range of skills within the group, and these can be either technical or soft skills. On the technical side there is an absolute requirement to be computer-literate, being comfortable working on the command-line in a Linux environment in the first instance and then building up knowledge and experience with all the built-in tools. Utilities such as the stream editor Sed and the data extraction utility Awk are invaluable in manipulating basic data files but, as per the Perl motto (TIMTOWTDI; pronounced ‘Tim Toady’), there is more than one way to do it. Following on from this is proficiency in at least one programming language; in my experience this is normally a scripting language such as Perl or Python, which as well as being easy to learn have vast repositories of (free) libraries available. The same can be said for other languages such as Java that offer object-orientated desktop and server-based applications. There is a tendency to perhaps ‘forget’ just how rich the landscape of libraries for software developers really is, and how much this has contributed to powerful scientific software.

Of course, as much biological and chemical information is stored in relational databases then knowledge of the Structured Query Language (SQL) is also a key skill for anyone routinely working with such information. Here again, the availability of free database systems (see above) provides the opportunity for anyone to explore and teach themselves without fear of breaking someone else’s database. Free SQL clients such as SQuirrel [33], Oracle SQL Developer [34] and in the MySQL world, MySQL Workbench [35] and PhpMyAdmin [36], provide many capabilities for zero cost. Again, the availability of powerful database management software is incredibly enabling and is another factor aiding companies whose primary goal is science, rather than software development.

Finally, data analysis is also a key skill. More and more, a good understanding of the basics of statistics will help understand the validity and outcome of the many large-scale analyses now available within biology. As such, knowledge of tools, particularly the R project for statistical computing, as well as bioinformatics applications, can be crucial. If more advanced machine-learning approaches are required, the Weka [37] system offers an extensive range of algorithms and techniques, often more than enough to test an initial hypothesis.

10.9 Conclusion

FLOSS fulfils many critical roles within OGT that would otherwise require investment in alternative commercial products. In selecting FLOSS over these other solutions, there is, of course, a cost element to be considered, but that is not the whole story. If any FLOSS product did not meet the required functional criteria then it would not be used; similarly, if a better commercial product becomes available then this would be considered. However, besides reducing expenditure on licences, there are additional advantages from taking a FLOSS approach.

First, if there is a problem, then the very nature of open source software means that, assuming you have the technical ability, it is possible to see directly where the problem is. This was experienced directly at OGT when the format of some NCBI Genbank records changed. The downstream effect of this was that an application using a BioJava file parser threw an error. Because the code was available it was possible to find the problem and fix it locally so the parsing could complete. But beyond that the fix could be passed to the developers for incorporation into the distribution and made available to the whole community. In the event of a problem that we are not able to resolve ourselves then there are usually many other users who can offer a solution or make suggestions on how to proceed. Encountering an error with a commercial product typically requires registration of the error and waiting for someone to do something about it. Even then you may have to wait for the next release before the fix is available to implement.

Similarly, by having access to the source code, there is the opportunity to adapt or extend the functionality to individual requirements as they may arise. In contrast, the functionality of a commercial product is decided by the manufacturers and any additions to this are decided by them alone. FLOSS also avoids the vagaries of a commercial manufacturer, for instance they may choose to amalgamate distinct products into a single offering. This then forces the users to use this new product if they want to keep using the most up-to-date version of the software. We have had experience of this with one piece of software that was incorporated into a more enterprising solution. Interestingly, after a couple of intermediate releases, the standalone product is now being made available. Similarly, commercial vendors control support for older versions of their products and decide when they will stop supporting older releases. Users are powerless to prevent this and helpless to its effects, which can mean they are forced to upgrade even when they would not choose to do so.

In conclusion, FLOSS has significant market share, and performs as well as commercial products with increased flexibility and adaptability. As such, I foresee it retaining a central role in OGTs ’omics platform.

10.10 Acknowledgements

I would like to thank Volker Brenner and Daniel Swan at OGT for reviewing and their helpful suggestions. Also, I would like to thank Lee Harland for his valuable editorial suggestions on the submitted manuscript.

10.11 References