Chapter 16
VisIt: An End-User Tool for
Visualizing and Analyzing Very Large
Data
Hank Childs
Lawrence Berkeley National Laboratory
Eric Brugger, Brad Whitlock
Lawrence Livermore National Laboratory
Jeremy Meredith, Sean Ahern, David Pugmire
Oak Ridge National Laboratory
Kathleen Biagas, Mark Miller, Cyrus Harrison
Lawrence Livermore National Laboratory
Gunther H. Weber, Hari Krishnan
Lawrence Berkeley National Laboratory
Thomas Fogal, Allen Sanderson
University of Utah
Christoph Garth
Technische Universit¨at Kaiserslautern
E. Wes Bethel, David Camp, Oliver R¨ubel
Lawrence Berkeley National Laboratory
Marc Durant, Jean M. Favre, Paul Navr´atil
Tech-X Corporation, Swiss National Supercomputing Center, Texas Advanced
Computing Center
16.1 Introduction ...................................................... 358
16.2 Focal Points ...................................................... 359
16.2.1 Enable Data Understanding ............................. 359
16.2.2 Support for Large Data ................................. 360
16.2.3 Provide a Robust and Usable Product for End Users .. 360
357
358 High Performance Visualization
16.3 Design ............................................................ 360
16.3.1 Architecture ............................................. 361
16.3.2 Parallelism ............................................... 362
16.3.3 User Interface Concepts and Extensibility .............. 363
16.3.4 The Size and Breadth of VisIt .......................... 364
16.4 Successes ......................................................... 364
16.4.1 Scalability Successes ..................................... 364
16.4.2 A Repository for Large Data Algorithms ............... 365
16.4.3 Supercomputing Research Performed with VisIt ........ 366
16.4.4 User Successes ........................................... 366
16.5 Future Challenges ................................................ 368
16.6 Conclusion ........................................................ 368
References .......................................................... 369
VisIt is a popular open source tool for visualizing and analyzing data. It owes
its success to its foci of increasing data understanding, large data support, and
providing a robust and usable product, as well as its underlying design that fits
today’s supercomputing landscape. This chapter, which draws heavily from a
publication at the SciDAC Conference in 2011 by Childs et al. [2], describes
the VisIt project and its accomplishments.
16.1 Introduction
A dozen years ago, when the VisIt project started, a new high performance
computing environment was emerging. Ever increasing numbers of end users
were running simulations and generating large data. This rapidly growing
number of large data sets prevented visualization experts from being inti-
mately involved in the visualization process; it was necessary to put tools in
the end users’ hands. Almost all end users were sitting in front of high-end
desktop machines with powerful graphics cards. But their simulations were
being run on remote, parallel machines and generating data sets too large to
be transferred back to these desktops. Worse, these data sets were too large
to even process on their (serial) machines anyways. The types of visualiza-
tion and analysis users wanted to perform varied greatly; users needed many
techniques for understanding diverse types of data, with use cases ranging
from confirming that a simulation was running smoothly to communicating
the results of a simulation to a larger audience, to gaining insight via data
exploration.
VisIt was developed in response to these emerging needs. It was (and is)
an open source project for visualizing and analyzing extremely large data
sets. The project has evolved around three focal points: (1) enabling data
understanding, (2) scalable support for extremely large data, and (3) providing
a robust and usable product for end users.
VisIt: An End-User Tool for Visualizing and Analyzing Very Large Data 359
In turn, these focal points have made VisIt a very popular tool for visu-
alizing and analyzing the data sets generated on the world’s largest super-
computers. VisIt received a 2005 R&D 100 award for the tool’s capabilities in
understanding large data sets. It has been downloaded hundreds of thousands
of times, and it is used all over the world.
16.2 Focal Points
16.2.1 Enable Data Understanding
In many ways, “VisIt” is a misnomer, as the name implies the tool is
strictly about visualization and making pretty pictures. The prospect of lost
name recognition makes renaming the tool unpalatable, but it is worthwhile
to emphasize that VisIt focuses on five primary use cases:
1. Visual exploration: users apply a variety of visualization algorithms to
“see” what is in their data.
2. Debugging: users apply algorithms to find a “needle in a haystack,” for
example, such as hot spots in a scalar field or cells that have become
twisted over time. The user then asks for debugging information in a
representation that is recognizable to their simulation (e.g., cell X in
computation domain D has a NaN).
3. Quantitative analysis: users apply quantitative capabilities ranging from
simple operations, such as integrating densities over a region to find its
mass, to highly sophisticated operations, such as adding a synthetic
diagnostic to compare to experimental data.
4. Comparative analysis: users compare two related simulations, two time
slices from a single simulation, simulation and experiment, etc. The tax-
onomy of comparative analysis has three major branches, each of which
is available in VisIt: image-level comparisons place things side-by-side
and has the user detect differences visually. Data-level comparisons put
multiple fields onto the same mesh, for example, to create a new field
for further analysis that contains the difference in temperature between
two simulations. Topological-level comparisons detect features in the
data sets and then allow those features to be compared.
5. Communication: users communicate properties of their data to a large
audience. This may be via movies, via images that are inserted into a
PowerPoint presentation, or via line plots or histograms that are placed
into a journal article.
360 High Performance Visualization
16.2.2 Support for Large Data
Twelve years ago, “large data” meant several hundred million cells. Today,
“large” means several hundred billion cells. In both cases, the definition of
“large” was relative to the resources for processing the data. And this is the
target for the VisIt project: data whose full resolution cannot fit into primary
memory of a desktop machine. Of course, the amount of data to load varies
by situation. Can time slices be processed one at a time? How many variables
are needed in a given analysis? Is it necessary to load multiple members of an
ensemble simultaneously? For VisIt, the goal was to provide an infrastructure
that could support any of these use cases, and it primarily uses parallelism to
achieve this goal.
16.2.3 Provide a Robust and Usable Product for End Users
Enabling data understanding for large data is a daunting task requiring
a substantial investment. To amortize this cost, the project needed to be
delivered to many user communities, across both application areas and funding
groups.
The “one big tool” strategy provides benefits to both users and developers.
Compared to a smaller, tailored effort, users have access to more functionality
and better underlying algorithms for processing data. For developers, the core
infrastructure undergoes an economy of scale, where many developers can
collectively develop a superior core infrastructure than they would be able to
do independently.
But the “one big tool” approach has negative aspects as well. Their user
interface tends to provide an overly rich interface where users find many fea-
tures to be meaningless and simply view them as clutter. Further, developers
must deal with a less nimble code base where making functionality changes
sometimes leads to unexpectedly large coding efforts.
Further, delivering a product to a large end user community incurs signif-
icant cost in and of itself: the VisIt project has almost a thousand pages of
manuals, several thousand regression tests that run every night, a sophisti-
cated build process, and a variety of courses designed to teach people to how
to use the tool. It requires multiinstitutional coordination for release manage-
ment, for responses to user requests, and for software development. And, of
course, the source code itself must be well documented to reduce barriers to
entry for new developers.
The developers of the VisIt project decided to “go big”: to pay the costs
associated with large user and developer bases in the hopes of writing a tool
that would be usable by many and developed by many.
VisIt: An End-User Tool for Visualizing and Analyzing Very Large Data 361
FIGURE 16.1: Diagram of VisIt programs and their communication. Image
source: Childs et al., 2011 [2].
16.3 Design
This section describes several facets of VisIt’s design, including VisIt’s
architecture, its parallelism approach, and its user interface concepts.
16.3.1 Architecture
VisIt employs a client-server design, where both client and server are com-
posed of multiple programs (see Fig. 16.1). Client-side programs, typically
run on the user’s local desktop, are responsible for both user interface and
rendering, since interactivity is paramount for these activities. The client-side
programs are:
• gui: A graphical user interface built using the Qt widget set.
• cli: A command line user interface built using the Python language.
• viewer: A program responsible for the visual display of data.
• Custom, streamlined user interfaces can also be added to VisIt. The
interfaces can either complement the gui and cli or replace them alto-
gether.
Server-side programs, typically run on a remote supercomputer that can access
the user’s data in a parallel fashion, are responsible for processing data. The
server-side programs are:
• engine: The program that applies visualization and analysis algorithms
to large data sets using parallel processing.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.