%\VignetteDepends{}
%\VignetteIndexEntry{NetCDF Exercises}
%\VignettePackage{AdvancedR2011}

\documentclass{article}

\usepackage{BioconductorVignette}
\SweaveOpts{keep.source=TRUE,width=9,height=8,eps=FALSE,include=FALSE}

\title{Self-Study Exercises}
\author{Nishant Gopalakrishnan \\ Fred Hutchinson Cancer Research Center}
\date{17-18 February, 2011}


\begin{document}
\maketitle
<<setup, echo=FALSE>>=
options(width = 40)
@ 
\section{Introduction}

The genotype information for the GWAS study is provided to you in a comma
separated file snpData.csv. This data has been converted to NetCDF based
storage (with byte precision) and stored as a file snpData.nc. These files
are available in the extdata folder of the \Rpackage{AdvancedR2011Data} package.

The exercises that are included in this document are intended to 
\begin{itemize}
\item Introduce the snp data and explore some of the options in \R{} to read
data from the snpData.csv file.
\item Get familiar with the application programming interface provided by the 
\Rpackage{ncdf} package for accessing NetCDF files and create a script  to 
retrieve horizontal slices of the data from the NetCDF file snpData.nc.
\item Create a more useful function \Rfunction{getGWAScols} that access vertical
slices of data (snps for all samples) from the NetCDF file. This function will
be added to the \Rpackage{StudentGWAS} package that is being developed as this
course progresses. The function should be general enough to work with NetCDF
files of different sizes.
\item Create a unit test to verify that the \Rfunction{getGWAScols} function 
produces the correct output.
\end{itemize}


\begin{Ext} % exercise
The goal of this exercise is to explore some of the functionality available in 
\R{} to read large text files in an efficient manner by reading in smaller chunks
at a time.
\begin{itemize}
    \item The snp data is available as a comma separated file snpData.csv located 
    in the extdata folder of the \Rpackage{AdvancedR2011Data} package. Make use of 
    the function \Rfunction{system.file} to get a path to the snpData.csv file.
    \item Use the function \Rfunction{scan} to read in only the 601th
    row from the snpData.csv file.
\end{itemize}
\end{Ext}


\begin{Solution}

<<scanIntro>>=
library(AdvancedR2011Data)
pth <- system.file("extdata", "snpData.csv", package = "AdvancedR2011Data")
dat <- scan(pth, what = character(0), sep =",",
                               skip = 600, nlines = 1, quiet = TRUE)
@
\end{Solution}

\begin{Ext} % exercise
The goal of this exercise is to create a script to read in all the snps for 
sample 601 from the NetCDF file snpData.nc and verify that the values read 
from the NetCDF file match those in the file snpData.csv. Both files are provided 
in the extdata folder of the \Rpackage{AdvancedR2011Data} package and can be located 
using the \Rfunction{system.file} function. The snpData.nc file contains
genotype information for 1000 samples(rows),  113735 snps(columns) per sample.

\begin{itemize}
    \item Use the function \Rfunction{open.ncdf} to open the NetCDF file
    snpData.nc.
    \item Use the function \Rfunction{get.var.ncdf} to read in the data for all
    snps for the sample 601 in the file snpData.nc
    \item Use the \Rfunction{scan} function to read in the snp data for sample
    601 from the snpData.csv file and compare its performance with that using
    the \Rfunction{get.var.ncdf} function for the snpData.nc file. 
    The \Rfunction{system.time} function can be used to compare the performance 
    of the two functions.
    \item Check if the results are identical using the \Rfunction{identical}
    function in \R{}.
    \item Finally close the NetCDF file using \Rfunction{close.ncdf}
\end{itemize}
\end{Ext}


\begin{Solution}
<<rowAccess>>=
library(ncdf)
st <- system.time
ncFile <- system.file("extdata", "snpData.nc", package = "AdvancedR2011Data")
nc <- open.ncdf(ncFile, write = FALSE)
st(dat <- get.var.ncdf(nc, varid = "snpData", start= c(601,1), count = c(1, 113735)))
csvFile <- system.file("extdata", "snpData.csv", package = "AdvancedR2011Data")
st(origDat <- scan(csvFile, what = integer(0), sep =",", skip = 600, nlines = 1,
            quiet = TRUE))
identical(as.vector(dat), origDat)
invisible(close.ncdf(nc))
@
\end{Solution}


\begin{Ext} % exercise

For most of the software development in the remainder of the course, we will
need to access snp data for all the samples(i.e. along the columns of the
NetCDF file) in smaller chunks(couple of columns at a time) for block processing
of data. To make it convenient to develop code as we go along, a smaller subset 
of the data that we have looked at so far has been provided as a NetCDF file in
the extdata folder of the \Rpackage{StudentGWAS} package. The file 
small\_snpData.nc contains genotype information of 50 samples, 25 snips per
sample.

Our goal in this exercise is to develop a \Rfunction{getGWAScols} function that
retrieves the genotype information for all the samples for the range of snps
specified by the user. 

The function should take three arguments 
\begin{itemize}
    \item nc: An ncdf file pointer obtained by a call to the
    \Rfunction{open.ncdf}
    \item first: A single integer, the start index for the snpData (column) to 
    be returned
    \item last: A single integer, the end index for the snpData to be returned.
\end{itemize}

The function should return a \Robject{matrix} of data type \Robject{raw} having 
dimensions with number of rows equal to the number of samples in the 
small\_snpData.nc file and columns corresponding to the range of input specified
by the user. The function \Rfunction{as.raw} can be used to coerce the integer 
data type returned by \Rfunction{get.var.ncdf} to the raw data type.

Since the \Rfunction{getGWAScols} function is going to be included in a package 
and will likely encounter NetCDF files of varying sizes, the function has to be written 
with general usability in mind.(rather than hard coding number of rows/columns 
in the NetCDF file etc.) Additionally, suitable checks need to be put in place 
for the input arguments to make sure they are of the correct type and are within 
bounds with regard to the dimensions of the NetCDF file.

To help with the function development, a convenience function 
\Rfunction{getNcdfVarSummary} has been provided in the file utils.R
included in the \R{} folder of the \Rpackage{StudentGWAS} package. This function
 takes the  NetCDF file pointer returned by the function \Rfunction{open.ncdf} as
 input and returns a list. Elements dim1 and dim2 of this list returned correspond 
 to the  number of rows and number of columns in the NetCDF file respectively. The 
 \Rfunction{getNcdfVarSummary} can be pasted into your \R{} session or the
 contents of the file utils.R can be sourced into your session using the 
 \Rfunction{source} command for this exercise.
 \end{Ext}
 
\begin{Solution}

<<colAccess, eval = FALSE , keep.source = TRUE>>=
library(StudentGWAS)
getNcdfVarSummary <- StudentGWAS:::getNcdfVarSummary
getGWAScols <- function(nc, first, last = first)
{
    if (!is.numeric(first) || length(first) != 1 || is.na(first))
        stop("'first' must be a single integer")
    if (!is.numeric(last) || length(last) != 1 || is.na(last))
        stop("'last' must be a single integer")
    ncInfo <- getNcdfVarSummary(nc, "snpData")
    nrows <- ncInfo$dim1
    ncols <- ncInfo$dim2
    if (first < 1 || last < first || last > ncols)
        stop("we need to have 1 <= 'first' <= 'last' <= nb col in NetCDF file")
    dat <- get.var.ncdf(nc, varid = "snpData", start= c(1,first),
            count = c(nrows, last - first + 1))
    structure(as.raw(dat), dim = dim(dat))
}

pth <- system.file("extdata", "small_snpData.nc", package = "StudentGWAS")
nc <- open.ncdf(pth)
dat <- getGWAScols(nc, 1, 4)
invisible(close.ncdf(nc))
@
\end{Solution}


\begin{Ext} % exercise

An important part of the software development process is to include unit tests
to verify that the results returned by the functions that we have implemented 
are correct. This becomes extremely useful as we try to optimize code as the 
development of the package progresses.

The goal of this exercise is to develop a unit test function
\Rfunction{testGWAScols} that checks if the values returned by the
\Rfunction{getGWAScols} function are correct. The expected result for a call to
the function \Rfunction{getGWAScols} for all columns and samples in the file 
small\_snpData.nc has as been provided to you as a file
small\_snpData.rda in the data folder of the \Rpackage{StudentGWAS} package.
This data can be loaded into an \R{} session using the \Rfunction{load} function. 

\begin{itemize}
    \item Create a unit test function \Rfunction{testGWAScols} that takes no inputs
    \item Retrieve the genotype information for all the snips and samples in the
    file small\_snpData.nc using the \Rfunction{getGWAScols} function that we
    developed in our previous exercise.
    \item Load the target data from the extdata folder of the
    \Rpackage{StudentGWAS} package using the \Rfunction{load} function.
    \item Check if the results match using the \Rfunction{identical} function.
\end{itemize}

\end{Ext}
 
\begin{Solution}

<<unitTest, eval = FALSE, keep.source = TRUE>>=
library(ncdf)
testGWAScols <- function() 
{   
    pth <- system.file("extdata", "small_snpData.nc", package = "StudentGWAS")
    nc <-  open.ncdf(pth, write = FALSE)
    ncols <- getNcdfVarSummary(nc, "snpData")$dim2
    current <- getGWAScols(nc, 1, ncols)
    load(system.file("extdata", "small_snpData.rda", package = "StudentGWAS"))
    identical(current, small_snpData)  
}
testGWAScols()
@
\end{Solution}

\end{document}
