A Data Cleaning Example

For this particular example,

• the variables of interest are stored as key:value pairs and
• a single data cell could contain multiple (unknown) number of key:value pairs.

Basically, we want to convert input dataset on LHS to the output dataset on the RHS as illustrated in the graphic below -

The objective is to separate these key-value pairs and store the values in corresponding key columns.

The hadleyverse packages make this task a fairly simple one, especially tidyr, stringr and magrittr.

Survival Analysis - 2

In my previous post, I went over basics of survival analysis, that included estimating Kaplan-Meier estimate for a given time-to-event data. In this post, I'm exploring on Cox's proportional hazards model for survival data. KM estimator helps in figuring out whether survival function estimates for different groups are same or different. While survival models like Cox's proportional hazards model help in finding relationship between different covariates to the survival function.

Some basic notes about Cox model -

• It's a semi-parametric model, as the hazard function (risk per unit time) does not need to be specified.
• The proportional hazard condition states that the covariates are multiplicatively related to the hazard. (This assumption should always be checked after fitting a Cox model). 
• In case of categorical covariates, the hazard ratio (e.g. treatment vs no treatment) is constant and does not change over time. One can do away with this assumption by using extended Cox model which allows covariates to be dependent on time.
• The covariates are constant for each subject and do not vary over time.

(There's one-to-one mapping between hazard function and the survival function i.e. a specific hazard function uniquely determines the survival function and vice versa. Simple mathematical details on this relationship can be found on this wikipedia page.)

I'm using the same datasets (tongue dataset from package KMsurv and a simulated dataset using survsim) and set of packages as used in the previous post - OIsurv, dplyr, ggplot2 and broom .

Survival Analysis - 1

I recently was looking for methods to apply to time-to-event data and started exploring Survival Analysis Models. In this post, I'm exploring basic KM estimator. It is a nonparametric estimator of the survival function. There are couple of instances when the KM estimator comes in handy -

• When the survival time is censored
• Comparing survival function for different preassigned groups.

Below I'm computing KM estimator for a real dataset (on time to death for 80 males who were diagnosed with different types of tongue cancer, from package KMsurv) and a simulated dataset (created using package survsim). In addition I am using survival, OIsurv, dplyr, ggplot2 and broom for this analysis. The first example is taken from an openintro tutorial.

The rmarkdown document illustrating below analysis can also be found here. In my future posts, I'm planning to explore more on following survival models -

• Proportional hazards model
• Accelerated failure time Model
• Multiple events model (More than 2 possible events)
• Recurring events (Each subject can experience an event multiple times).

ogdindiar: R package to easily access Open Government Data from India Portal

Following up on my earlier posts on accessing Open Government Data from R, I've wrapped this code into an R package - ogdindiar. It's available on GitHub at https://github.com/steadyfish/ogdindiar

It provides one simple function - fetch_data() to download required data resource from the https://data.gov.in portal. You can find the details about the usage in this vignette.

Below is an example that downloads India's annual and seasonal mean temperature data using this package. You can also see it here.

dplyr Use Cases: Non-Interactive Mode

The current release of dplyr (v 0.4.1) offers lot more flexibility regarding usage of important verbs in non-interactive mode. In this post, I'm exploring different possible use-cases.

• group_by_, select_, rename_:

For group_by_, select_ and rename_, we can pass a character vector of variable names as an argument to .dots parameter.

• filter_:

To use filter_ function, we need to pass filter criteria as a parameter to .dots. The criteria can be created using lazyeval::interp function.

• mutate_, transmute_, summarise_:

We need to provide 2 things to these functions - a list of functions to be applied on the input variables (with corresponding input variables) and a character vector of output variables names. These 2 things can be passed to the .dots argument using combination of lazyeval::interp  and setNames function.

• joins:

For 2 table verbs, there's no *_join_ function and we don't need one for general purposes. We can just pass a named vector to by argument. setNames function comes in handy while doing this.

The R Code for the above mentioned use cases is shown below and can also be found on this GitHub Gist.

	# using dplyr finctions in non-interactive mode
	# examples
	library(plyr)
	library(dplyr)
	d1 = data_frame(x = seq(1,20),y = rep(1:10,2),z = rep(1:5,4))
	head(d1)
	#### single table verbs ####
	# group_by
	group_by_fn <- function(d_in,gp_vec){
	d_out = d_in %>%
	group_by_(.dots = gp_vec)
	}
	gp_vec = c("y","z")
	d1_gp_by_out = group_by_fn(d1,gp_vec)
	head(d1_gp_by_out)
	# select/rename (haven't included drop variables case)
	select_fn <- function(d_in,sel_vec){
	d_out = d_in %>%
	select_(.dots = sel_vec)
	}
	sel_vec = c("x","y")
	d1_select_out = select_fn(d1,sel_vec)
	head(d1_select_out)
	# filter
	filter_fn <- function(d_in,filter_crit){
	d_out = d_in %>%
	filter_(filter_crit)
	}
	y_vec = 6:8
	filter_crit = interp(~ filter_var %in% y_vec,filter_var = as.name("y"))
	d1_filter_out = filter_fn(d1,filter_crit)
	head(d1_filter_out)
	z_vec = 1:2
	filter_crit2 = interp(~ filter_var1 %in% y_vec & filter_var2 %in% z_vec,.values = list(filter_var1 = as.name("y"),
	filter_var2 = as.name("z")))
	d1_filter2_out = filter_fn(d1,filter_crit2)
	head(d1_filter2_out)
	# mutate, transmute, summarise
	mutate_fn <- function(d_in,op_ls,var_vec){
	d_out = d_in %>%
	mutate_(.dots = setNames(op_ls,var_vec))
	}
	var1_rng = 3:5
	op_ls = list(interp(~f(var1,var2), .values = list(f = as.name("*"),
	var1 = as.name("x"),
	var2 = as.name("y"))),
	interp(~f(var1 %in% var1_rng,var2,var3),.values= list(f = as.name("ifelse"),
	var1 = as.name("x"),
	var2 = as.name("y"),
	var3 = as.name("z"))))
	var_vec = c("yy","zz")
	d1_mutate_out = mutate_fn(d1, op_ls, var_vec)
	head(d1_mutate_out)
	var_ls = list("yy","zz")
	d1_mutate_out1 = mutate_fn(d1, op_ls, var_ls)
	head(d1_mutate_out1)
	#### two table verbs ####
	# joins
	d2 = data_frame(xx = seq(1,20),yy = rep(1:10,2),zz = rep(1:2,10))
	join_fn <-function(d_in1,d_in2,var_vec1,var_vec2){
	d_out = d_in1 %>%
	left_join(d_in2,setNames(var_vec2,var_vec1))
	}
	var_vec1 = c("x","y")
	var_vec2 = c("xx","yy")
	d_join_out = join_fn(d1,d2,var_vec1,var_vec2)
	head(d_join_out)
	# everything combined (essentially, power of %>%)
	d_combined_out = d1 %>%
	filter_fn(filter_crit) %>%
	group_by_fn(gp_vec) %>%
	mutate_fn(op_ls,var_vec) %>%
	select_fn(c("x","y","z")) %>%
	join_fn(.,d2,var_vec1,var_vec2)
	head(d_combined_out)
	# sources:
	# http://cran.r-project.org/web/packages/dplyr/vignettes/nse.html
	# http://stackoverflow.com/questions/28125816/r-standard-evalation-for-join-dplyr

view raw dplyr_functions_programmatic_use.R hosted with ❤ by GitHub