# Stat 710 Lecture 1: Basic Data Types


# Course Mechanics

Course goals: Be able to write clean, accurate, maintainable
code. Learn how the algorithms you will use as a statistician work so
that you can diagnose problems and extend or tweak them to fit your
needs.


- First half of the course focuses on software engineering, some
specifics of R, and how to write good code.

- Second half of the course focuses on algorithms.

- Course will be primarily in R. One homework will be in python, and
you'll have the option of doing the final homework in python as well
if you would like more practice.

- Course website:
[jfukuyama.github.io/teaching/stat710](jfukuyama.github.io/teaching/stat710)
will have slides, assignments, and any additional readings.

- Homework submitted through canvas.

-----

Assessment:

- 40% homework. Homeworks will generally be posted on Sunday and due the following
Tuesday, 9 days later.

- 30% midterm. In class, written.

- 30% final exam. Like the midterm, at the scheduled final exam time.


-----

## Textbooks

Main text is Matloff, _The Art of R Programming_. _The R Cookbook_, by
Paul Teetor, will also be useful.

How to read the book:

- Matloff has example code, you should have an R session open and actually type in the commands, make sure that you get the same results.

- Don't worry about the analogies to C (unless you have a lot of experience in C and you find that helpful!)

- You can generally skip the extended examples.

-----

## Reading for today

Reading: Matloff Chapter 1, 2, 3.1, 3.2, 3.4, 3.5, 3.6, 3.7, 4.1, 4.2, 4.3, 5.1, 5.2 , 6.1

Chapter 1: Don't worry about understanding everything in detail, but it gives a good overview and will help you have a slot for everything later.

Also, you should probably not use his advice on getting help, the real way is to google your problem + R and click on the first stackoverflow link.

It's almost never a good idea to ask for help on the mailing list.

# Data types and data structures

Goal today: Learn about the ways R stores data and how to access and
manipulate data in those structures.

Why is this important?

- R is object-oriented and functional (we'll discuss this more later
in the course).

- This sometimes makes it seem like magic: the `plot` function does
completely different things in different contexts. How does R know?


Data types for today:

- Vectors

- Matrices

- Lists

-----


# Vectors

Fundamental data type in R, scalars are actually just vectors of length 1.

## Creating a vector

Easiest way to make a vector is with the `c` function (for "concatenate"):

```{r}
x = c(1, 168, .3)
x
```

----

The `:` operator will also make a vector of numbers separated by 1:
```{r}
y = 5:10
y
```

What do you think `y = 5.1:10` will do?

----

`seq` makes a vector containing units with arbitrary spacing:

```{r}
seq(from = 12, to = 30, by = 3)
```

----

You can specify the length you want instead of the spacing:

```{r}
seq(from = 1.1, to = 2, length = 10)
```

----

`rep` will give a vector of repeated elements:

```{r}
rep(1, times = 5)
```

----

You can repeat vectors instead of single numbers:

```{r}
rep(c(1, 5, 7), times = 3)
```

-----

Specifying `each` instead of `times` changes how the repeats work: the
first element in the vector is repeated `each` times, then the second,
then the third:
```{r}
rep(c(1, 5, 7), each = 3)
```

-----


## Some vector operations

Vectors can be added, subtracted, multiplied, etc.

If the vectors are the same size, this will be done element-wise.

```{r}
x = c(1, 2, 4)
x + c(5, 0, -1)
x * c(5, 0, -1)
x / c(5, 4, -1)
```

-----

What if the vectors are not the same size?

Remember that scalars in R are just vectors, so in the following code
we are actually adding two vectors:

```{r}
x = c(1, 2, 4)
x + 2
```

-----

## Vector recycling

_This is weird and important!_

If a vector operation requires the vectors to be the same length, R
automatically repeats the shorter one until it is long enough to match
the longer one.

That is what happened in the example above: the longer vector, `x`,
had length 3, and the shorter vector, `2`, had length 1. What really
happened was more like

```{r}
x = c(1, 2, 4)
x + rep(2, times = 3)
x + 2 # notice that these give the same results
```


-----

This makes sense for adding a vector and a scalar (= vector of length
1).

It can give results you're not expecting if you have vectors of
different lengths:

```{r}
c(1, 2, 4) + c(6, 0, 9, 20, 22)
## same as
c(1, 2, 4, 1, 2) + c(6, 0, 9, 20, 22)
```

If the vector lengths are not multiples of one another R will warn you
about the recycling. Otherwise, it does it without complaint:

```{r}
c(2, 3, 7, 8) + c(0, 1)
## same as
c(2, 3, 7, 8) + rep(c(0, 1), times = 2)
```

-----

## Logical operations

`>`, `<`, `<=`, `>=`, `==` are all vector operations and work like the
arithmetic operators:

```{r}
c(2, 5, 7) > c(1, 3, 8)
c(2, 5, 7) < 8
```

-----

`all` and `any` work on boolean vectors (vectors of `TRUE` and
`FALSE`), and do what is implied by their name:

```{r}
any(c(TRUE, FALSE))
all(c(TRUE, FALSE))
x = 1:10
any(x > 8)
any(x > 88)
all(x > 88)
all(x > 0)
```

-----

## Testing vector equality

We can't use `==` to test whether vectors are the same because `==`
will give us a boolean vector.

```{r}
x = 1:3
y = c(1, 3, 4)
x == y
```

----

Two ways around this `all` and `identical`:

```{r}
all(x == y)
identical(x, y)
```

. . .

You need to be a bit careful with identical though:

```{r}
identical(1:3, c(1, 2, 3))
## why?
typeof(1:3)
typeof(c(1, 2, 3))
```

-----

## Vector access

To get a subvector, we can use `[]`.

The format is `vector1[vector2]`, where `vector2` gives the indices we
want to pick out of `vector1`.

```{r}
y = c(1.2, 3.9, 0.4, 0.12)
y[c(1,3)]  # extract elements 1 and 3 of y
y[2:3]
v = 3:4
y[v]
```

----

We are allowed to repeat indices:

```{r}
y
y[rep(c(1, 3), each = 3)]
```

----

To exclude indices instead of include them, we can use `-`:

```{r}
z = c(5, 12, 13)
z[-1]  # exclude element 1
z[-1:-2]  # exclude elements 1 through 2
```

-----

## Filtering

We will often want a sub-vector, but we won't know in advance which
elements we want.

As a contrived example, suppose we want to extract just the elements
of the vector whose square is greater than 8.

We could use the following code to do so:

```{r}
z = c(5, 2, -3, 8)
w = z[z * z > 8]
w
```

-----

. . .

What's happening here? 

```{r}
z
z * z
z * z > 8
z[c(TRUE, FALSE, TRUE, TRUE)]
z[z * z > 8]
```

-----

Another simple way to extract only some of the values of a vector is with `subset`:
```{r}
z = c(5, 2, -3, 8)
z[z * z > 8]
subset(z, z * z > 8) ## gives the same result
```

-----

We can also use `which`. `which` just gives us the positions at which
the condition occurs, and we can use those positions to get the
relevant subvector:

```{r}
z = c(5, 2, -3, 8)
which(z * z > 8)
z[which(z * z > 8)]
```

----

# Matrices

Matrices in R turn out to just be vectors with an additional attribute
giving the dimensions.

-----

## Matrix creation

Most basic way of creating a matrix is with the `matrix` function.

It takes a vector giving the values that should go in the matrix plus
number of rows and number of columns.

By default, the values are arranged in column-major order, but you can
specify that the data is coming in in row-major order instead:

```{r}
y = matrix(c(1, 2, 3, 4), nrow = 2, ncol = 2)
y
y = matrix(c(1, 2, 3, 4), nrow = 2, ncol = 2, byrow = TRUE)
y
```

Note that `byrow = TRUE` doesn't change the way the matrix is stored
behind the scenes.

----

You can specify just a number of rows or just a number of columns, and
R will figure out what the dimensions of the matrix should be for you:
```{r}
y = matrix(c(1, 2, 3, 4), nrow = 2)
y
```

-----

What we learned about vector recycling applies to matrices too:

```{r}
y = matrix(1:2, nrow = 2, ncol = 5)
y
y = matrix(1:3, nrow = 2, ncol = 5)
y
```

What happens with `matrix(1:3, nrow = 2)`?

-----

## Matrix operations

These work pretty much the same way vector operations do.

With matrices we get matrix multiplication, `%*%` in addition to the
operations available for vectors.

```{r}
y = matrix(c(1, 2, 3, 4), nrow = 2)
y
y %*% y
y * 3
y + y
```

-----

## Matrix indexing


Getting sub-matrices is analogous to getting sub-vectors.

We still use `[]`, but now we have two indices.

The syntax is `matrix[rowIndices, colIndices]`, where `rowIndices` is
a vector describing the rows you want to take and `colIndices` is a
vector describing the columns you want to take.

Leaving one empty means that you want to take all the available
indices.

-----

. . .

Let's see how this works:

```{r}
z = matrix(sample(1:12), nrow = 4)
z
z[,2:3] ## extract the 2nd and 3rd columns
z[2:3, 2] ## extract the 2nd and 3rd rows of the 2nd column
z[,-1] ## negative subscripts work the same way as with vectors
```

-----

We can assign values to submatrices using this indexing as well:

```{r}
y = matrix(1:6, nrow = 3)
y
y[c(1,3),] = matrix(c(1, 1, 8, 12), nrow = 2)
y
```

-----

## Filtering

As with vectors, we can filter to submatrices by generating indices
that we want to keep.

As a simple example, suppose we want just the rows of the matrix for
which the second column is at least 3.

We could do that as follows:

```{r}
x = matrix(c(1, 2, 3, 2, 3, 4), nrow = 3)
x
x[x[,2] >= 3, ]
```

-----

. . .

What is happening?

```{r}
x
x[,2]
x[,2] >= 3
x[c(FALSE, TRUE, TRUE), ]
x[x[,2] >= 3, ]
```

-----

## Avoiding unintended dimension reduction


Notice what happens if we try to make a sub-matrix corresponding to
just one row of `x`:

```{r}
x
r = x[2,]
r
```

What happened?


-----

```{r}
attributes(x)
attributes(r)
str(x)
str(r)
```

`r` is a vector!

-----

`drop = FALSE` lets us avoid this behavior.

```{r}
r = x[2,,drop = FALSE]
r
attributes(r)
str(r)
```

# Lists

- Lists are technically vectors.

- The vectors from before were called _atomic_ vectors, which means
that their components couldn't be broken down into smaller components.

- In R, the main purpose of lists is to lump together data of
different types. Atomic vectors require all their elements to be of
the same type, but lists can have any sort of elements (including lists!).

----

## Creating lists

You can create a list with the `list` function...

```{r}
j = list(name = "Joe", salary = 55000, union = TRUE)
j
```

-----

The component names (_tags_) are not necessary, but it is good
practice to use them.

```{r}
list("Joe", 55000, TRUE)
```

-----

## List access

Suppose we want to get Joe's salary. There are at least three
different ways to do so:

```{r}
j$salary
j[["salary"]]
j[[2]]
```

The double brackets `[[]]` allow us access to an element of the list.

Note that if we didn't use the tags, we would only be able to access
the salary using `j[[2]]`.

----

. . .

## Sublist vs. Element of list


Very important: `[[]]` and `[]` are different operations:

- `[[]]` gives an _element_ of the list

- `[]` gives a sublist

```{r}
j[[2]]
class(j[[2]])
j[2]
class(j[2])
```

-----

## List manipulation

We can add to a list by tag:

```{r}
j$hobby = "sailing"
j
```

-----

We can add to a list by index:

```{r}
j[[5]] = 1:5
j
```

----

We can remove an element from a list by setting it to `NULL`:

```{r}
j[[5]] = NULL
j
j$hobby = NULL
j
```

-----


## Extracting names and values

Names are easy, just use the `names` function:

```{r}
names(j)
```

-----



To get values, we use `unlist`.

This is another function you have to be careful with, because it
doesn't necessarily do what you think it will.

- It returns an atomic vector.

- That means all the elements have to be of the same class, and so
data often gets coerced to a different type.

- If we unlist `j`, we get back a character vector. This is
essentially because R knows how to convert numbers and booleans to
characters but doesn't know how to convert characters to numbers or
booleans.

```{r}
unlist(j)
```

You can see the text for the coercion hierarchy.

----

Note that `unlist` gives an atomic vector even for recursive lists
(lists of lists), for example:

```{r}
complex_list = list(a = list(1), b = 1:5, c = (list(a = 1, b = 2)))
unlist(complex_list)
```





