--- title: "Transformations and statistical representations for images in R" author: "Brian B. Avants" date: "`r Sys.Date()`" bibliography: REFERENCES.bib output: rmarkdown::html_vignette vignette: > %\VignetteEngine{knitr::rmarkdown} %\VignetteIndexEntry{Transformations and statistical representations for images in R} %\VignetteEncoding{UTF-8} \usepackage[utf8]{inputenc} --- ```{r global options, include=FALSE} library(knitr) library(ANTsR) runLongExamples<-FALSE setANTsDeterministic(FALSE) ``` > "A small leak will sink a great ship." (folk wisdom) # Introduction The ANTs*R* package interfaces state of the art image processing with *R* statistical methods. The project grew out of the need, at University of Pennsylvania, to develop large-scale analytics pipelines that track provenance from scanner to scientific study. ANTs*R* achieves this by wrapping an ANTs and ITK C++ core via `Rcpp` [@dirksbook]. [ITK](http://www.itk.org/) is a templated C++ framework with I/O and support for arbitrary image types (usually 2, 3 or 4 dimensions) as well as surface representations. [ANTs](http://stnava.github.io/ANTs), built on ITK, focuses on multivariate image matching and segmentation as well as geometric (even high-dimensional) image transformation. Both tools are [deeply validated and widely used](http://journal.frontiersin.org/ResearchTopic/1580). Together, these tools allow powerful image manipulations. However, they lack a true statistical back-end. Historically, statistical software was not amenable to direct manipulation of multiple dimensional images. This led to "in-house" statistical programming or, perhaps worse (for science), reliance on closed source commercial software. Given the increasing [popularity of *R*](http://r4stats.com/articles/popularity/) and prominence of quantitative imaging, it is natural that *R* should have a package focused on biological or medical image analysis. This package integrates several frameworks for extracting quantitative information from images and mapping images into reference coordinate systems. Human brain mapping studies have long relied on Talairach-Tournoux and related coordinate systems [@TALAIRACH1958]. Similar standardized localization is becoming more common within non-human studies [@Johnson2010;@Majka2013]. Atlases of other organ systems are also emerging and being applied clinically [@deMarvao2014]. This class of methods relies on *image transformation* and *image segmentation* as an aid to the ultimate goal of quantifying variability within and across populations. Longer term, such methods will be critical to individualized patient care and other translational applications. ## ANTs*R* Algorithms Here, we provide an overview of the methods available within ANTs*R*. - core image processing and I/O: ITK [@Avants2014a]; - registration and utilities for image processing: ANTs mappings [@Tustison2014] and feature extraction [@Tustison2014a]; - dimensionality reduction: Eigenanatomy [@Dhillon2014] and SCCAN [@Avants2014]; - methods for ASL-based cerebral blood flow quantification [@Kandel2015]; - neighborhood representations of images that enable rich statistical models [@Kandel2015] - core statistics and temporal filtering via *R* packages that is amenable to BOLD image processing In combination, these tools enable one to go from near-raw medical imaging data to a fully reproducible scientific publication [@Avants2015]. ## Data organization and access in ANTs*R* This package uses an `antsImage` S4 class to hold pointers to ITK images. We convert `antsImage` objects to *R* objects before passing them to *R* statistical methods. E.g. we convert a _scalar_ image to a vector, a collection of scalar images to a matrix or a time series image to a matrix. Currently, ANTs*R* does not explicitly represent images with vector valued voxels (e.g. tensor or warp images) although these may be supported in the future in a way that is similar to our current support for time series images. The large majority of images employed within ANTs*R* are of 2, 3 or 4 dimensions with `float` pixel types. This information is stored within the `antsImage` class. A few example images are built into ANTs*R*, but more can be downloaded. See `?getANTsRData`. ```{r basic read,message=FALSE,warnings=FALSE} img<-antsImageRead( getANTsRData("r16"), 2 ) # built in image img ``` Take a quick look at the image. ```{r basic_plot,message=FALSE,warnings=FALSE,echo=FALSE} img2<-antsImageRead( getANTsRData("r64"), 2 ) # built in image invisible(plot(img2)) ``` ## Contributions of the package ANTs*R* includes: - An organizational system such that relatively small scripts may implement full studies - Implementation of foundational methods + Smoothing, temporal filtering, etc + functional image denoising via `compcor` and `*DenoiseR` + flexible: easy to estimate voxel-wise statistical models - Reference simulation data and examples distributed with the package - Interpretation of results + sparse low-dimensional predictors + anatomical labeling of predictors based on AAL and other coordinate systems - Openness and reproducibility In total, ANTs*R* is a rigorous framework upon which one may build customized statistical implementations appropriate for large-scale functional, structural or combined functional and structural image analyses. Because much of the core is implemented with C++, the framework also remains efficient. Finally, note that `Rscript` allows one to send ANTs*R* scripts to clusters and take advantage of distributed computing resources. ## Basic ANTs*R* functionality Here, we quickly summarize ANTsR functionality and useful tools. **The travis build system** We test ANTs*R* regularly. The status of the build (and an expected build result) can be seen here: [![Build Status](https://travis-ci.org/stnava/ANTsR.png?branch=master)](https://travis-ci.org/stnava/ANTsR). Take a look at the detailed log to see what one might expect if building ANTs*R* from source. **Image input and output** If nothing else, ANTs*R* makes it easy to read and write (medical) images and to map them into a format compatible with *R*. Formats we frequently use include jpg, tiff, mha, nii.gz and nrrd. However, only the last three have a proper physical space representation necessary for mapping. Below is an example of how we access this type of image and see its geometry. Check `antsImageWrite` for the primary supported I/O. ```{r physical space} mnifilename<-getANTsRData("r27") img<-antsImageRead(mnifilename, pixeltype="unsigned char") img # retval<-antsImageWrite(img,mnifilename) antsGetSpacing(img) antsGetDirection(img) antsGetOrigin(img) print(img[120,122]) # same type of thing in 3 or 4D print(max(img)) ``` **Index an image with a label** Often, you would like to summarize or extract information from within a known region of an image with arbitrary shape but within a given intensity "zone". We simulate this below and show a few accessors and type conversions. ```{r indexing} gaussimg<-array( data=rnorm(125), dim=c(5,5,5) ) arrayimg<-array( data=(1:125), dim=c(5,5,5) ) img<-as.antsImage( arrayimg ) print( max(img) ) print( mean(img[ img > 50 ])) # if using SUBSET using an antsImage, you must be explicit sub = as.array(img >= 50) > 0 print( mean( gaussimg[ sub ]) ) ``` **Convert a 4D image to a matrix** Four dimensional images are generated and used in the same way. One can easily transform from 4D image to matrix and back. ```{r fourd} gaussimg<-makeImage(c(5,5,5,10), voxval = rnorm(125*10) ) print(dim(gaussimg)) avg3d<-ANTsR::getAverageOfTimeSeries( gaussimg ) mask <- avg3d < 0.25 gmat<-timeseries2matrix( gaussimg, mask ) print(dim(gmat)) ``` If one has a mask, then one can use `makeImage` to generate a new image from a scalar or vector. ```{r makeimage} newimg<-makeImage( mask, mean(avg3d) ) # from scalar newimg<-makeImage( mask, colMeans(gmat) ) # from vector ``` **Convert a list of images to a matrix** Often, one has several scalar images that need to be accumulated for statistical processing. Here, we generate a simulated set of these images and then proceed to smooth them, store them in a list and convert them to a matrix after extracting the information of each image within a data-driven mask. ```{r imagelist} nimages<-100 ilist<-list() for ( i in 1:nimages ) { simimg<-makeImage( c(50,50) , rnorm(2500) ) simimg<-smoothImage(simimg,1.5) ilist[[ i ]] = simimg } # get a mask from the first image mask<-getMask( ilist[[1]], lowThresh=mean(ilist[[1]]), cleanup=TRUE ) mat<-imageListToMatrix( ilist, mask ) print(dim(mat)) ``` Once we have a matrix representation of our population, we might run a quick voxel-wise regression within the mask. Then we look at some summary statistics. ```{r image matrix} mat<-imageListToMatrix( ilist, mask ) age<-rnorm( nrow(mat) ) # simulated age gender<-rep( c("F","M"), nrow(mat)/2 ) # simulated gender # this creates "real" but noisy effects to detect mat<-mat*(age^2+rnorm(nrow(mat))) mydf = data.frame( age = age, gender=factor(gender )) mdl<-lm( mat ~ age + gender, data=mydf ) mdli<-bigLMStats( mdl, 1.e-4 ) print(names(mdli)) print(rownames(mdli$beta.t)) print(paste("age",min(p.adjust(mdli$beta.pval[1,])))) print(paste("gen",min(p.adjust(mdli$beta.pval[2,])))) ``` **Write out a statistical map** We might also write out the images so that we can save them for later or look at them with other software. ```{r write betas} agebetas<-makeImage( mask , mdli$beta.t[1,] ) returnval<-antsImageWrite( agebetas, tempfile(fileext ='.nii.gz') ) ``` ## More ANTs*R* functionality We achieve quantification in biological or medical imaging by using prior knowledge about the image content. **Segmentation** In segmentation, we assume the image has a known set of tissues, organs etc. Here, we assume 3 tissues exist and use a classic k-means model with MRF penalty [@Avants2011]. Note that we also bias correct the image to help it match our model [@Tustison2010]. ```{r segmentation} fi<-antsImageRead( getANTsRData("r16") ,2) fi<-n3BiasFieldCorrection(fi,2) seg<-kmeansSegmentation( fi, 3 ) invisible(plot(seg$segmentation)) ``` If you like segmentation, also look at `rfSegmentation` and `atropos`. **Registration** In registration, we assume the image can be mapped to some canonical shape or example, i.e. an atlas. Or to another individual. ANTs*R* provides a simple wrapper for SyN image registration [@Tustison2013], ```{r syn,echo=TRUE,message=FALSE,warnings=FALSE} fi<-antsImageRead( getANTsRData("r16") ) mi<-antsImageRead( getANTsRData("r64") ) mytx<-antsRegistration(fixed=fi , moving=mi , typeofTransform = c('SyN')) regresult<-iMath(mytx$warpedmovout,"Normalize") fiedge<-iMath(fi,"Canny",1,5,12) invisible(plot(regresult, list(fiedge), window.overlay=c(0.5,1)) ) ``` while `invariantImageSimilarity` provides powerful multi-start search for lower dimensional affine registrations. Deformable image registration results in a voxel-wise map of the contraction and expansion of the moving image (after affine transformation) that is needed to map to the fixed image. This deformation gradient is colloquially known as "the jacobian". ```{r jacobian,echo=TRUE,message=FALSE,warnings=FALSE} jac<-createJacobianDeterminantImage(fi,mytx$fwdtransforms[[1]],1) invisible(plot(jac)) ``` Above, we compute and plot the image of the log-jacobian. This mapping is a useful summary measurement for morphometry [@Avants2012,@Kim2008]. **Registration and segmentation** Registration and segmentation are often applied jointly or iteratively to maximize some criterion. See the example in `jointIntensityFusion` for one such case [@Wang2013a]. **Neighborhood operations** Basic I/O and management of images as vectors is critical. However, there is additional information that can be gained by representing an image and its neighborhood information. ANTs*R* represents image neighborhoods, which capture shape and texture, as a matrix. Here, extract an image neighborhood matrix representation such that we may analyze it at a given scale. ```{r nhood} mnit<-getANTsRData("r16") mnit<-antsImageRead( mnit ) mnit <- resampleImage( mnit , rep(4, mnit@dimension) ) # downsample mask2<-getMask(mnit,lowThresh=mean(mnit),cleanup=TRUE) radius <- rep(2,mnit@dimension) mat2<-getNeighborhoodInMask(mnit, mask2, radius, physical.coordinates = FALSE, boundary.condition = "mean" ) print(dim(mat2)) ``` The variable `mat2` has size determined by the neighborhood radius (here, 5) and the number of non-zero voxels in the mask. The `boundary.condition` says how to treat data that is outside of the mask or the image boundaries. This example replaces missing data with the mean in-mask value of the local neighborhood. Other useful tools in ANTs*R* include `iMath`, `thresholdImage`, `quantifyCBF`, `preprocessfMRI`, `aslPerfusion`, `computeDVARS`, `getROIValues`, `hemodynamicRF`, `makeGraph`, `matrixToImages`, `rfSegmentation`, `antsRegistration`, `plotPrettyGraph`, `plotBasicNetwork`, `getTemplateCoordinates`, `antsSet*`. Several image mathematics operations (like `ImageMath` in ANTs) are accessible too via `iMath`. ## Example label sets and data ANTs*R* also provides AAL label [@Tzourio-Mazoyer2002] names via: ```{r aal} data(aal,package='ANTsR') labs<-1:90 ``` with cortical labs defined by `labs`. The DKT atlas labels are similarly summarized in `DesikanKillianyTourville` [@Klein2012]. An example BOLD correlation matrix is available in `bold_correlation_matrix`. This can be used to try out `makeGraph` and related functions. ## Visualization and plotting The basic `plot` function is implemented for the `antsImage` class. It can show 2 or 3D data with color overlays, the latter of which can display multiple slices side by side. Several color choices are available for the overlays. For 3D images, see `renderSurfaceFunction` and `plotBasicNetwork` for `rgl` and `misc3d` based interactive surface and network plots. Another such example is in `visualizeBlob`. These are too long-running to compile into the vignette but the help examples for these functions will allow you to see their results. A good visualization alternative outside of ANTs*R* is [antsSurf](https://github.com/stnava/antsSurf). ## BOLD data processing with ANTs*R* Good approaches exist in ANTs*R* for preprocessing BOLD data. These yield both motion matrices and relevant summary measurements such as FD and DVARS. See `?preprocessfMRI` for a simplified utility function. This function could be used on each run of an experiment and the results stored in organized fashion for later use. **Motion correction** To motion correct your data, one might run: ```{r motioncorr,results='hide',eval=FALSE} # get an average image averageImage <- getAverageOfTimeSeries( boldImage ) motionCorrectionResults <- antsMotionCalculation( boldImage, fixed = averageImage ) ``` A `moreaccurate` flag should be set to `1` or `2` for usable (not test) results. FD and DVARS are returned which may be used to summarize motion. One might also get this data from `preprocessfMRI` which also provides denoising options based on data-driven methods including frequency filtering. For more fMRI focused tools, see [RKRNS](http://stnava.github.io/RKRNS/) and its github site [github RKRNS](https://github.com/stnava/RKRNS). ## Dimensionality reduction Images often have many voxels ($p$-voxels) and, in medical applications, this means that $p>n$ or even $p>>n$, where $n$ is the number of subjects. Therefore, we often want to "intelligently" reduce the dimensionality of the data. We favor methods related to PCA and CCA but have a few ICA related tools too. **Eigenanatomy & SCCAN** Our sparse and geometrically constrained dimensionality reduction methods seek to both explain variance and also yield interpretable, spatially localized pseudo-eigenvectors [@Kandel2014a,@Cook2014]. This is the point of "eigenanatomy" which is a variation of sparse PCA that uses (optionally) biologically-motivated smoothness, locality or sparsity constraints. ```{r eanat,echo=TRUE,message=FALSE,warning=FALSE,eval=FALSE} # assume you ran the population example above eanat<-sparseDecom( mat, mask, 0.2, 5, cthresh=2, its=2 ) eanatimages = matrixToImages( eanat$eig, mask ) eseg<-eigSeg(mask, eanatimages ,F) jeanat<-joinEigenanatomy(mat, mask, eanatimages, c(0.1)) eseg2<-eigSeg(mask,jeanat$fusedlist,F) ``` The parameters for the example above are set for fast processing. You can see our paper for some theory on these methods [@Kandel2014a]. A more realistic study setup would be ```{r eanatreal,eval=FALSE} eanat<-sparseDecom( inmatrix=mat, inmask=famask, nvecs=50, sparseness=0.005, cthresh=500, its=5, mycoption=0 ) jeanat<-joinEigenanatomy( mat , famask, eanat$eig, c(1:20)/100.0 , joinMethod='multilevel' ) useeig<-eanat$eig useeig<-jeanat$fusedlist avgmat<-abs(imageListToMatrix( useeig , famask )) avgmat<-avgmat/rowSums(abs(avgmat)) imgmat<-( mat %*% t(avgmat) ) ``` The `imgmat` variable would be your summary predictors entered into `lm` or `randomForest`. More information is available within the examples that can be seen within the help for `sparseDecom`, `sparseDecom2` and the helper function `initializeEigenanatomy`. **Sparse canonical correlation analysis** CCA maximizes $PearsonCorrelation( XW^T, ZY^T )$ where $X, W$ are as above and $Z$ and $Y$ are similarly defined. CCA optimizes the matrices $W, Y$ operating on $X, Z$ to find a low-dimensional representation of the data pair $( X , Z )$ in which correlation is maximal. Following ideas outlined in @Dhillon2014 and @Avants2014, this method can be extended with sparsity constraints that yield rows of $W, Y$ with a controllable number of non-zero entries. See the [sccan tutorial](http://stnava.github.io/sccanTutorial/) and `sparseDecom2` for more information. ## Conclusions With the current ANTs*R*, one may: - Exploit ANTs and ITK functionality within *R* - Leverage *R* functionality to help understand and interpret imaging data - Use feature selection based on various filtering strategies in `iMath` and elsewhere (e.g `segmentShapeFromImage`) - Employ dimensionality reduction through eigenanatomy or SCCAN with a variety of incarnations, some of which are similar to ICA - Use relatively few interpretable and low-dimensional predictors derived from high-dimensional data. - Interpret multivariate results intuitively when used in combination with standard *R* visualization. See [ANTsR](https://github.com/stnava/ANTsR) for all source code and documentation and [RKRNS-talk](http://stnava.github.io/RKRNS) for html slides that discuss extensions to BOLD decoding. Enjoy and please refer issues to [ANTs*R* issues](https://github.com/stnava/ANTsR/issues). # References