A Unified Framework for Neuroscience Morphological Data Visualization

Abstract

The complexity of the human brain makes its understanding one of the biggest challenges that science is currently confronting. Due to its complexity, the brain has been studied at many different levels and from many disciplines and points of view, using a diversity of techniques for getting meaningful data at each specific level and perspective, producing sometimes data that are difficult to integrate. In order to advance understanding of the brain, scientists need new tools that can speed up this analysis process and that can facilitate integrating research results from different disciplines and techniques. Visualization has proved to be useful in the analysis of complex data, and this paper focuses on the design of visualization solutions adapted to the specific problems posed by brain research. In this paper, we propose a unified framework that allows the integration of specific tools to work together in a coordinated manner in a multiview environment, displaying information at different levels of abstraction and combining schematic and realistic representations. The two use cases presented here illustrate the capability of this approach for providing a visual environment that supports the exploration of the brain at all its organizational levels.

1. Introduction

Understanding the human brain is one of the greatest challenges still open. There are a number of factors that limit progress in this field. The first one is the brain’s overwhelming complexity, which comes from the combination of a set of factors: First, the number of significant components is huge (around ${10}^{11}$ neurons and ${10}^{15}$ synapses in the human brain [1]); furthermore, other elements such as glia and vasculature are also relevant. Second, the morphology of each of these elements is far from trivial, as many details that determine important aspects of their behavior exist. Third, morphology alone is not enough, since it is necessary to understand also each element’s function. Fourth, the brain is not just a collection of complex elements working on their own, but a network of elements, and the network topology is also central for understanding how the brain works. Finally, it is necessary to consider not only a stationary network; its dynamic behavior is also central for understanding the brain’s behavior and how it adapts to changing stimuli.

In order to cope with this complexity, the brain has been studied at many different levels and from many disciplines and points of view (morphology, physiology, etc.), using a diversity of techniques for getting meaningful data at each specific level and perspective. Despite the task’s difficulty, interest in understanding the brain can be tracked back to the very beginning of science, among other reasons, because of its influence on human healthy life. The improvement of laboratory equipment and techniques, the availability of powerful computers and dedicated software, and the appearance of ambitious collaborative research programs has strongly accelerated the pace at which new advances are being produced. Nevertheless, there are still many issues where progress is needed, such as in understanding the brain as a unified system that spans multiple levels of organization (covering all levels from the molecular to the behavior and cognitive).

The human brain can be seen as a complex processing system that can be structured into different organizational levels, allowing it to be studied at different scales, among other reasons, because of the availability of techniques that can provide information at each specific scale. This approach has provided a good knowledge about each of the individual levels, but there is still a lack of understanding about the interactions and causal chains of events among the different levels of the hierarchy needed to result in cognitive and behavioral processes. One of the reasons for this is the lack of tools that facilitate moving smoothly across the hierarchy; with respect to this, one of the goals of this paper is to present a visualization approach that can help in bridging this gap by facilitating users to combine data and descriptions from different levels of abstraction and standpoints (restricted so far to morphology and physiology, to soon include connectivity).

Presenting information across the hierarchy of levels at which the brain can be studied is not easy. The available knowledge and information is by no means comprehensive; there are still many missing pieces in the puzzle, which makes smooth transitions difficult. Going down in the hierarchy often means increasing the volume of data by orders of magnitude. This stands in the way of a well-established principle for information visualization: overview first, zoom and filter, then details-on-demand [2]), since getting one level down in the hierarchical description of a region easily results in cluttering the visualization with huge amounts of information that often prevents getting a global view of the data being analyzed.

This paper presents a framework for the visual exploration of neuroscience data that can facilitate the analysis of complex scenarios, based on a multilevel approach designed to follow closely the brain organizational levels which are relevant for a particular task. In order to remove clutter from visualizations, this framework uses schematic or iconic symbols that have higher expressive capabilities than realistic (morphologically correct) representations of the relevant entities. Being based on the possibility of combining schematic representations of higher order abstractions with other lower level descriptions, this method can effectively decrease the visual complexity of the scenes presented to the user. This, in turn, facilitates the analysis task in extensive scenarios, where other visual representations would be cluttered with unnecessary details. Last, this framework takes into consideration many of the complexity aspects that hamper neuroscience data analysis, proposing a visual approach that has proved to be helpful for the exploration and understanding of massive data and scenarios. A preliminary implementation is also presented, focused on the morphological aspects of neuronal circuits within the brain cortex and on certain features of the dendritic spines of cortical pyramidal neurons. This gives rise to an integrative framework that combines different tools, with complementary approaches that work in a coordinated manner to provide a multiview interactive environment in which it is possible to explore data at different levels of abstraction while applying different representation techniques.

The paper is organized as follows. After this introduction and a brief review of previous work, the main aspects of the proposed visualization framework are described, together with two case studies: a first application for the multilevel navigation around medium to high complexity neocortex neuronal circuits and a second one for the analysis of the morphological features of dendritic spines in pyramidal neurons, also within the neocortex. These two use cases work at different scales, from cortical columns to dendritic spines. The difficulties associated to the visualization of each of them are different and require distinct techniques to address their respective challenges. However, the proposed solution allows the combination of specialized tools, providing a single framework which allows transversing the entire range of scales, linking the knowledge obtained from each of them and, therefore, facilitating advancing toward the unraveling of the complete system. Last, some illustrative results are described and discussed.

Related Work

Visualization, the set of techniques that use computer-generated graphics and representations to facilitate human understanding, has attracted a great amount of interest during the last years [3,4,5], being a discipline that is quickly reaching maturity. Graphical representations of data are used to pursue the goal of presenting the information aesthetically while providing insight into the depicted data. Related to this goal, disciplines such as data or information visualization are becoming especially relevant for the analysis of systems that are either highly complex or follow the 3Vs rule (volume, velocity, and variety) that characterizes big data [6]. The strategies behind most visualization techniques rely on our ability to process information through the visual channel, which is definitely more efficient in detecting patterns or outstanding elements from graphical representations than from numerical or tabular representations [7,8].

The increasing complexity of the data sets that can be displayed thanks to the continuous advances in visualization technologies has led to the use of multiscale visualization methods in a myriad of different fields, ranging from rechargeable metal batteries [9] to molecular dynamics simulations [10,11] or geomorphology [12], to cite just a few. Apart from that, there have been different attempts to use coordinated linked views for providing users with a more complete way to gain insight into such complex data sets [13,14,15].

Regarding the neuroscience domain, the complexity of the brain makes its study a challenging job that requires the collaboration of different disciplines. On one side, there is the extraction of activity data [16]. On the other side, different software tools based on graphical representations of the data have been successfully used in a wide variety of problems, such as the exploration of neuronal data (Connectome Explorer [17], Brain Gazer [18], or Neuron Navigator [19]); segmentation and tracing tasks (NeuroMorphoVis [20], Imaris [21] or Neurolucida [22]), and reconstruction, visualization, or simulation (NeuroConstruct [23], Neuronize [24,25], NeuroTessMesh [26], NeuroScheme [27], RTNeuron [28], VIOLA [29] or ViSimpl [30]), among many others.

However, the extreme complexity of the brain and its multilevel organization require innovative techniques to describe and analyze each organizational level, placing more emphasis on the most relevant features under study and less on the least significant ones, while relating different scales and domains in order to fully explore the system (for example, for correlating anatomical and functional features). In this sense, alternative strategies have already been applied through the design of abstract or iconic representations applied to neural circuits and electrical models [31,32,33,34,35,36,37], or the inclusion of nonphotorealistic rendering techniques [38,39,40]. Some authors have designed multiscale abstract visualization techniques for the analysis of neurites and their connections, such as [41].

It should be noted that the work presented here goes one step beyond, illustrating the potential of innovative visualization strategies based on the systematic combination of accurate (realistic) and abstract (symbolic or iconic) representations, together with transformed domains. This combination of different kinds of representations with transformed domains should be (and is) finely tuned to the specific range of tasks to be performed, which usually involve the exploration of sets of neuroscience data within a multiscale framework, tightly coupled with the organizational levels of the brain.

2. Visualization in Neuroscience

There are many areas in neuroscience that can benefit from the availability of tools designed for helping researchers perform typical analysis operations (such as characterize, compare, select, model, predict, hypothesize, validate, etc.). For example, in a large collaborative framework such as the Human Brain Project, there are research groups working on neuron morphology, neuronal circuit simulation, neuromorphic hardware design, neuroinspired robotics, etc. Usually, many of the analysis operations in these fields have to be performed on extensive data sets, being impossible for anyone to get a global overview together with the fine detail needed at a specific region unless the analysis system provides the possibility of combining representations of different scale or level of detail. This paper presents a visualization approach based on the combination of symbolic and realistic views of different expressive power, designed specifically for this purpose.

Figure 1 and Figure 2 present a series of sketches that summarize the main idea proposed in this paper. For example, let us suppose that a user wants to analyze certain morphological features of the spines located in a dendrite fragment of a pyramidal cell, within a large collection of neurons. Figure 1a–c illustrate the sequence of navigation steps followed by the user in order to select that neuron’s specific area of interest. The spines of the selected dendrite segment are presented afterwards, ranked according to the user selected features in Figure 2a–c, using a color code to enhance the variation of the desired morphological features. These sketches highlight two points:

• The advantage of using abstract representations, which allows condensing large portions of complex geometry in order to simplify the representation of a scenario, composed in this case by a potentially large number of neurons within a specific brain area.
• The benefits obtained by using transformed domains. In this case, representing the spines extracted out of the dendrite, ranked by the value of the features under analysis, helps during the analysis stage by providing users with a clear overview of the spines’ morphological variability (an ordered set of elements is easier to understand, characterize, and compare than a randomly disordered set).

Specifically, the framework presented here merges three tools that cooperatively enable the combination of abstract and realistic representations spanning the organizational levels of the cerebral cortex, from the scale of cortical columns to dendritic spines: NeuroScheme, NeuroTessMesh, and PyramidalExplorer. Neuroscheme provides the visualization of abstract representations at different levels of abstraction, as well as useful functionalities for filtering, sorting, and selecting elements. These operations can be performed at any level of abstraction, being automatically propagated to the rest of the organizational levels. The second tool, NeuroTessMesh, is oriented to the efficient visualization of neuronal anatomy, rendering the morphology of cells at different levels of detail. This tool provides realistic representations of neurons, allowing the (interactive, under conditions) visualization of complex cortical circuits thanks to the on-the-fly refinement of the polygonal mesh that approximates the neuronal membrane. Last, PyramidalExplorer has been designed for the visual exploration and discovery of the detailed structure of the cell microanatomy. This tool supports the examination of the dendritic spines of a neuronal cell and provides useful analysis techniques, such as filtering, sorting, or content-based retrieval operations.

These tools are integrated into the framework and can share data interaction operations such as selection, filtering, or camera movements in coordinated views. For example, filtering out a cortical layer in NeuroScheme can be automatically propagated to NeuroTessMesh, so that all the morphologies of the neurons belonging to that layer disappear from the realistic view.

The next section shows the results of applying the proposed framework on two practical examples involving the visual exploration of complex data sets.

3. Results

The proposed framework has been used to perform two different tasks of exploratory data analysis of neural data. Each of them addresses the visualization of data sets at different levels of abstraction and requires specific tools that can collaborate within the proposed framework, thus providing an environment that supports the visual analysis of neural morphology at different organizational scales.

3.1. Case Study 1: Navigation Around the Neocortex

The first case study addresses the visual exploration of a set of cortical neurons. This task can be performed using the multilevel schematic cell depiction features provided by NeuroScheme, a tool for multilevel navigation around the neocortex designed for facilitating the analysis of the morphology, topology, and simulation results of relatively complex synthetic neocortical neuronal circuits (up to a few million neurons) [27]. For that purpose, this method uses a multilevel structure where each level corresponds to a different neocortical organizational level. In this case, the entities that compose the multilevel structure are: column, microcolumn, neuron, neurite, and (neurite) branch entities, with each of them having a different symbolic representation (the first three ones are presented in Figure 3). The selection of any of them involves the selection of all of the lower level elements included in the entity. As commented before, mixing visualizations of data and structures at different levels allows users to gain access to the global picture and the desired fine detail when needed, facilitating also the selection of the part of the circuit to be displayed at each moment. Neuroscheme can be combined with NeuroTessMesh, in order to show a coordinated view with the detailed geometry of the neurons (Figure 4).

The geometrical and the abstract representations provided here are connected, so that a selection or modification in the abstract representation is automatically updated in the geometrical representation. For example, focusing at the neuronal level, if a neuroscientist is interested in knowing the characteristics of the neurons included in a circuit, their schematic representation allows him to see at a glance the main circuit’s neuron types and their most relevant features. Additionally, the schematic neuron representations can be ranked according to different criteria (such as the value of specific features, like soma volume or dendritic length), or they can be displayed occupying their real position within the circuit. After a set of neurons is ranked, groups of neurons can be selected for studying them in more detail or for analyzing their detailed morphology and circuit location. It has to be noted that, just like the geometrical and abstract windows, the organizational levels are also connected. A selection in one level, such as the selection of a group of neurons at the neuron level, results in displaying only the minicolumns containing the selected neurons when going one level up in the hierarchy. Whenever a column or microcolumn layer is selected, only the neurons from that layer are highlighted in the abstract and geometrical windows and when going down in the hierarchy. Figure 5 and Figure 6 show images of a column and a microcolumn; please note that the column symbolic representation has been expanded to show the microcolumns that compose it.

Regarding the symbolic representations for each organizational level, the highest one in the hierarchy is the column, being represented by a blue pentagonal shape containing the average information of the neurons belonging to it (which are depicted according to the neuron entity, described below). A column can be expanded to view the most relevant information of each of its layers, showing both the number or pyramidal cells and the number or interneurons per layer. For each layer, this number is represented using the length of a red bar growing leftward for pyramidal cells and rightward for interneurons. The triangles or circles drawn next to each layer allow selecting all the pyramidal or interneuron cells of that column layer. A triangle (or a circle) painted in gray represents that, for that column, there are no pyramidal cells (or interneurons) in that particular layer.

The next organizational level down in the hierarchy corresponds to microcolumns. They are represented by green hexagonal shapes, containing also information about the average neurons in that microcolumn. Just like before, microcolumns can be expanded to inspect layer information. Going another step down the hierarchy, neurons are represented by circles, having their color encode whether it is an excitatory neuron (blue circle) or inhibitory neuron (red circle). At the neuron center, a triangle in the pictogram denotes that it refers to a pyramidal neuron, and a circle denotes that it refers to an interneuron. Other symbols show degree of dendritic arborization and soma and dendrite area and volume.

The lowest two levels in the hierarchy correspond to the neurite and branch entities. Just like before, relevant information is graphically presented, letting users select also how much neurite detail is presented. Additional details can be found in [27].

3.2. Case Study 2: Analysis of Dendritic Spines in Pyramidal Neurons in the Neocortex

The second case of study shows the potential of mapping different quantitative features across the neurons’ morphological representation. This option facilitates the study of the features’ spatial distribution, bridging also the gap between the bare statistical analysis of feature distributions and the performance of visual analysis processes, which allow studying spatial distributions and detecting the occurrence of patterns in the features’ spacial distribution.

This case of study is based on PyramidalExplorer [42], an interactive tool designed for exploring and analyzing the organization and architecture of pyramidal neurons. It should be noted that previously existing tools like Connectome Explorer [17] study aspects such as if an axon connects with the same dendrite or with different dendrites, as well as whether synapses are sequential or not within the same dendrite. By contrast, PyramidalExplorer allows the detection of possible regional differences through the combination of implemented functional models and structural quantitative morphological information. The tool accepts as input different data, such as electron microscopic reconstructions or confocal microscopy stacks of high-resolution images. Based on these input data, a model of the neuron is obtained. Figure 7 presents the 3D reconstruction of a human pyramidal neuron. In this case, the goal is to analyze the volume of the spines along the dendrites. For that, the tool provides the following widgets with the aim of:

• Obtaining a list with all the dendritic spines. The spines in this list can be ordered according to different criteria.
• Allowing the selection of analysis criteria by interacting with the set of considered spine characteristics.
• Allowing a 3D visualization of the details of each individual spine within the neuron.
• Using the histogram distribution to filter the results and allowing to select the most adequate color range in order to improve the results visualization.

For the same pyramidal neuron, Figure 8 shows how different morphological characteristics are distributed along the whole neuron, which allows the detection of possible differences in the regions of the neuron architecture. The extracted morphological features are spine length, spine area, spine volume, spine neck mean diameter, and spine maximum diameter.

The selection of the minimum spine neck mean diameter and the maximum spine diameter, for example, makes it easy to spot where are the spines that have the narrowest necks with the largest heads (Figure 8a). As can be seen, the distribution of these spines in the arbor differs from those with other characteristics (Figure 8b). Additionally, the tool allows linking other types of information to each spine. For example, it is possible to combine functional models (generated offline) with morphology structural quantitative information to visualize a membrane potential peak estimation for each spine. Figure 8c) shows how the membrane potential peak values are distributed along the neuron. Differences in specific dendritic segments, even within the same dendrite, can be easily spotted.

Last, PyramidalExplorer could be used in combination with other tools, for example, for the visual analysis of functional information, which requires also the availability of neuron anatomical details.

4. Discussion

The two use cases presented in this article are just two examples that illustrate different tasks for the exploratory analysis of neuronal data. The first one deals with the navigation across a visually complex set of neurons, where complexity comes both from the number of cells that compose it (in the order of thousands) and from the intricate anatomy of the neurons themselves. The visualization of a neuronal forest composed of hundreds or thousands of cells yields cluttered, visually complex images where it is difficult to distinguish one element from another. In this context, the use of abstract representations that simplify neuron anatomy decreases drastically visual clutter, displaying the most relevant information while eliminating minor details. Additionally, structuring these schematic representations at different levels of abstraction significantly simplifies the amount of information associated with the different organizational levels of the brain, taking therefore advantage of the benefits of encapsulation when analyzing complex systems. Overall, the combination of schematic representations with the exploitation of the levels of abstraction concept makes it possible to reduce both visual and cognitive complexity when visually exploring complex neural scenes.

However, the visualization of the real neuron anatomy is still necessary for studying their morphological features. For that reason, the approach presented here combines schematic representation at different levels of abstraction with the visualization of morphologically correct representations. This generates clean and compact views that are efficiently rendered at rates suitable for interactivity. Additionally, this approach provides combinations of schematic representations that provide overall views with more detailed data, obtained at lower levels of abstraction, when and where needed. For example, in Figure 4, it is easy to see that layer 6 in the column contains far more pyramidal neurons than interneurons. Since the abstraction levels are connected, a simple click on the red bar representing the number of interneurons would allow to display these cells instantly, descending to the minicolumn level or the cell level as desired. Furthermore, since the abstract view and the geometrically realistic view are also connected, once a particular object has been spotted in the abstract representation, clicking on it can automatically highlight it in the realistic view and zoom in for an easy location (Figure 4b).

Being able to go from abstract to realistic views provides many additional advantages, mainly because it lets users to select the kind of representation better suited to a particular task. For example, abstract views facilitate arranging data according to different criteria, since in these representations the objects spatial positions do not necessarily have to correspond to their real anatomical positions. This way, objects in the abstract image can be reordered according to size, volume, or any meaningful feature, facilitating filtering and ranking operations. After some elements are selected, it is always possible to display them in their original locations.

The proposed framework facilitates creating more powerful tools. For example, use cases 1 and 2 can be connected, so that the exploration of a region of the neocortex can provide a selection of pyramidal neurons from which the dendritic spines will be later on analyzed. In general, once certain particular objects have been spotted through the interaction with the tools, as in the first use case, the framework allows to go beyond and focus on one of them, descending to a lower level and combining information from different nature, as it happens in the second use case with functional and morphological information. This way, Figure 8 shows regional differences based on morphological information (spines’ neck diameter) and functional models (membrane potential).

However, it should be emphasized that complex experimental data sets are not yet available at present to enable the performance of exploration activities linking all levels of abstraction. In this sense, our prototype constitutes a pioneering proposal and its ability to cover all organizational levels of the brain sets the basis for the integration of the specific knowledge acquired at each of the scales. The tools integrated in the initial prototype are mainly focused on the visualization of anatomical information; in order to build a fully integrative framework, it will be necessary to incorporate techniques for the visualization of neural activity and connectivity data, thereby providing a tool that helps to link the different brain study domains.

5. Conclusions and Future Work

The analysis of neuroscience data will require, sooner rather than later, the availability of powerful analysis tools that can help cope with the continuous increase in complexity found in the data sets provided by new experimental and computational techniques and infrastructures. In this sense, using symbolic or schematic representations allows summarizing nonessential details, letting users concentrate on relevant information while analyzing massive scenarios.

Additionally, the establishment of large multidisciplinary research initiatives will demand the availability of tools that can integrate data and models stemming from different disciplines. In order to meet these demands, new representation models and techniques will be needed; these models, together with the development of appropriate analysis techniques, will allow researchers to gain insight into the brain as a unified system, even though it will be studied from many different perspectives.

The unified framework proposed in this paper, based on the design of symbolic representations and the use of transformed domains, has been developed having these two problems in mind. This framework also facilitates the integration of specific tools, by providing a multiview environment where they can be run in a coordinated manner. Using this approach, information provided by different techniques and experiments can be displayed coordinately, at different levels of abstraction, allowing users to navigate among them, while decreasing the level of visual clutter provided to users through the exploitation of schematic representations.

With respect to the methods presented here, the possibility of having schematic representations alleviates the amount of visual clutter provided to users, facilitating the generation of global views that summarize the most relevant information. At the same time, having this schematic representation linked to the morphologically detailed information allows the selection of objects to view in full detail in the 3D space whenever needed.

The representations described in this article, not necessarily linked to any particular discipline, can be designed for facilitating the fusion of information from different origins (such as morphological and functional, for instance), because their symbolic or abstract nature can facilitate combining information and data from multiple sources. Having a rich variety of representations at different scales can simplify condensing an otherwise overwhelming amount of detail. The early feedback received from end users, so far, is positive, since this kind of tools allows them to access to information that was unavailable until now, or because it facilitates analysis tasks by speeding up the navigation process around large data sets, allowing them also to combine information gathered at different scales.

Future work includes the application of these techniques to the navigation around large data sets and repositories, facilitating finding the localization of the desired information. An additional line of work is the development of new workflows for improving the productivity of neuroscientists by providing early insight on the data being collected.