---
title: "STATS 220"
subtitle: "Tidy data`r emo::ji('broom')`"
type: "lecture"
date: ""
output:
  xaringan::moon_reader:
    css: ["assets/remark.css"]
    lib_dir: libs
    nature:
      ratio: 16:9
      highlightStyle: github
      highlightLines: true
      countIncrementalSlides: false
---

background-image: url("https://images.unsplash.com/photo-1587654780291-39c9404d746b?ixlib=rb-1.2.1&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=1950&q=80")
background-size: 100% 100%

--

<img src = "https://images.unsplash.com/photo-1615465502839-71d5974f5087?ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&ixlib=rb-1.2.1&auto=format&fit=crop&w=1350&q=80", width = 45%, style = "position:absolute; top: 22.5%; left: 5.5%; box-shadow: 3px 5px 3px 1px #00000080;"></img>

--

<img src = "https://images.unsplash.com/photo-1615750198206-6e632e147ef9?ixlib=rb-1.2.1&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=1350&q=80", width = 45%, style = "position:absolute; top: 22.5%; left: 52.5%; box-shadow: 3px 5px 3px 1px #00000080;"></img>

.green[.footnote[image sources: Upsplash]]

???

* each piece is a scalar, a vector, a tibble, a list
* each piece is vector functions or table functions
* bc they are standard and consistent blocks,
* ensemble them to a simple mario, or complex nintendo
* so far you worked with several datasets now,
* hopefully you find a pattern is using these dplyr verbs in different ways to solve various problems.

---

background-image: url("img/what-you-think-about-ds.png")
background-size: contain

```{r initial, echo = FALSE, cache = FALSE, results = 'hide'}
library(knitr)
options(htmltools.dir.version = FALSE, htmltools.preserve.raw = FALSE, 
  tibble.width = 60, tibble.print_min = 6)
opts_chunk$set(
  echo = TRUE, warning = FALSE, message = FALSE, comment = "#>",
  fig.path = 'figure/', cache.path = 'cache/', cache = TRUE, fig.retina = 3,
  fig.align = 'center', fig.width = 4.5, fig.height = 4, fig.show = 'hold',
  dpi = 120
)
```

```{r xaringan-panelset, echo = FALSE}
xaringanExtra::use_panelset()
```

```{r external, include = FALSE, cache = FALSE}
read_chunk('R/06-tidy-data.R')
```

## .yellow[What you think about data science] `r emo::ji("star_struck")`

.footnote[image credit: Hilary Parker]

???

* 80% on modelling

---

background-image: url("img/what-ds-in-reality.png")
background-size: contain

## .yellow[What data science is in reality] `r emo::ji("sob")`

.footnote[image credit: Hilary Parker]

???

* 80% on cleaning
* it's invaluable to know the data pipeline: how data collected and organised
* covid19, different def for each country in the early stage

---

## .center[tidy data <i class="fas fa-not-equal"></i> clean data]

.brown[.center[The `movies` data is tidy but not clean.]]

```{r movies, echo = FALSE, highlight.output = 6:7}
```

???

* all data you've seen so far are tidy, but tidy doesn't mean clean.
* how I would clean this data: 
* either google Drei release year, and corresponding GDP
* or replace GDP with NA
* Gross are suspicious

---

## .center[tidy data <i class="fas fa-not-equal"></i> clean data]

.brown[.center[They are tidy and clean in their own way.]]

.pull-left[
* `time_use`
```{r timeuse, echo = FALSE}
```
]
.pull-right[
* `countrycode`
```{r country, echo = FALSE}
```
]

???

* week 4, when we try to join them, we see how country names are encoded differently.
* Data cleaning is laborious, and highly dependent on the data itself.
* No golden standards or principles to guide how to clean data.
* reason about the cleaning steps (intuition, common sense, business context)
* principles to get tidy data.

---

class: middle inverse

# Tidy data describes a standard way of storing data in .blue[a consistent format].

???

* tidy data is about data format, instead of caring the data values.

---

## Your turn

Tuberculosis data from WHO (`data/tb.csv`):
1. Is `tb` tidy data?
2. What are the data observations? What are the data variables?

```r
tb
```

```{r tb, echo = FALSE}
```

`r countdown::countdown(minutes = 1, seconds = 0, font_size = "48px", top = 0)`

???

* Can you easily plot a dataset like this using ggplot2? by sex and age
* come back later

---

class: center

## one data, many representations

<img src="https://github.com/gadenbuie/tidyexplain/raw/master/images/static/png/original-dfs-tidy.png" height="520px">

.left[.footnote[image credit: Garrick Aden-Buie]]

???

* which rep does `tb` resemble?
* which rep is easier to work with? depends on the question, wide -> suitable for summary table

---

background-image: url(https://github.com/allisonhorst/stats-illustrations/raw/master/rstats-artwork/tidydata_1.jpg)
background-size: contain

???

* In database, this is schema.
* Tidy data principles are a rephrase of third norm in a database schema design.
<https://en.wikipedia.org/wiki/Third_normal_form>, to data scientists.
* tidy data is for human consumption.
* Tabular data is column-oriented format

---

background-image: url(https://github.com/allisonhorst/stats-illustrations/raw/master/rstats-artwork/tidydata_2.jpg)
background-size: contain

---

background-image: url(https://github.com/allisonhorst/stats-illustrations/raw/master/rstats-artwork/tidydata_5.jpg)
background-size: contain

## Making friends with tidy data

* one set of consistent tools for different datasets
* easier for automation and iteration

.footnote[reference: Illustrations from the Openscapes blog [Tidy Data for reproducibility, efficiency, and collaboration](https://www.openscapes.org/blog/2020/10/12/tidy-data/) by Julia Lowndes and Allison Horst]

???

* To work with messy data, every time you need to switch to different gears
* learn new tools that just works for that specific dataset
* It's much more pleasant to work with tidy data, help you to build a good taste of data analysis
* spent less flighting with different tools, focus more on data analysis bc one set of consistent tools
* the {tidyverse} philosophy to work with the tidy data structures
* build automatic workflow for analysis, feed different data sets.

---

## .center[Get data into tidy format]

<br>
.pull-right[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="240px">](http://tidyr.tidyverse.org)]
]
.pull-left[
```{r tidyr-gt, echo = FALSE}
library(gt)
tidyr <- tribble(
  ~ type, ~ `fun1()`, ~ `fun2()`,
  "**pivoting**", "`pivot_longer()`", "`pivot_wider()`",
  "**splitting/combining**", "`seperate()`", "`unite()`",
  "**nesting/unnesting**", "`nest()`", "`unnest()`",
  "**missing**", "`complete()`", "`fill()`"
)

tidyr %>% 
  gt() %>% 
  fmt_markdown(everything()) %>% 
  cols_label(
    `fun1()` = md("`function()`"),
    `fun2()` = md("`function()`")
  ) %>% 
  as_raw_html()
```
]

???

* missing verbs not exactly a pair

---

.pull-left[
.x[
* Messy
]
<hr>
```{r tb-2, echo = FALSE, highlight.output = 2}
print(tb, width = 36)
```
]

.pull-right[
.checked[
* Tidy
]
<hr>
```{r tb-fill, echo = FALSE, highlight.output = 2}
```
]

???

* the `tb` data is messy data, rows and columns
* obs and variables
* working with tidy data, we know obs: the number of cases by sex and age groups over years at country level
* messy in a way that column names should be values, stored in cells.
* what steps needed to get to this tidy format.

---

.left-column[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="60%">](http://tidyr.tidyverse.org)]
### - pivot
]
.right-column[
* `pivot_longer()` a wider-format data
```{r tb-pivot, highlight.output = 1:2}
```
]

???

* messy: store two variables into one column 

---

.left-column[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="60%">](http://tidyr.tidyverse.org)]
### - pivot
]
.right-column[
<br>
.center[![](img/tidyr-longer-wider.gif)]
]

.footnote[image credit: Garrick Aden-Buie & Mara Averick]

---

.left-column[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="60%">](http://tidyr.tidyverse.org)]
### - pivot
### - split
]
.right-column[
* `seperate()` a character column into multiple columns
```{r tb-sep}
```
]

---

.left-column[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="60%">](http://tidyr.tidyverse.org)]
### - pivot
### - split
### - fill
]
.right-column[
* `fill()` in `NA` with previous (.brown[`"down"`]) or next (.brown[`"up"`]) value
```{r tb-fill, echo = 1}
```
]

???

* how to impute missing data is dependent on data context. For cross-sectional, average and build a regression model to predict
* fill by direction
* demo on `direction`

---

background-image:url(img/lst-col.png)
background-position: 85% 50%
background-size: 10%

.left-column[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="60%">](http://tidyr.tidyverse.org)]
### - pivot
### - split
### - fill
### - nest
]
.right-column[
* `nest()` multiple columns into a .brown[list-column]
```{r tb-nest}
```
]

???

* tibble is a list, recursive structure
* we can put a list into a tibble
* complex objs in tibble

---

## Building many models*

```{r tb-mod, echo = 1}
```

???

* r2 for each country

---

.panelset[

.panel[.panel-name[Plot]


```{r tb-mod-plot, echo = FALSE, fig.width = 9, fig.height = 4.5}
```

.panel[.panel-name[Code]

```{r ref.label = "tb-mod-plot", eval = FALSE}
```
]
]
]

???

* direct labeling instead of > 200 country labels underneath

---

## Your turn

Auckland weather data from GHCND (`data/ghcnd/ghcnd-akl.csv`):

1. Is `aklweather` tidy data?
2. What are the data observations? What are the data variables?

```{r aklweather, echo = 2}
```

`r countdown::countdown(minutes = 1, seconds = 0, font_size = "48px", top = 0)`

???

* The `value` column contains multiple types of measurements.

---

.pull-left[
.x[
* Messy
]
<hr>
```{r ref.label = "aklweather", echo = FALSE, highlight.output = 2}
```
```{r ref.label = "aklweather", eval = FALSE}
```
]

.pull-right[
.checked[
* Tidy
]
<hr>
```{r akl-wide, echo = FALSE, highlight.output = 2}
```
]

???

* demo on `read_csv2()`
* daily weather information in Auckland
* temp values unbelievably large?

---

.left-column[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="60%">](http://tidyr.tidyverse.org)]
### - calibrate
]
.right-column[
* calibrate the measurements
```{r akl-clean}
```
]

---

.left-column[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="60%">](http://tidyr.tidyverse.org)]
### - calibrate
### - pivot
]
.right-column[
* `pivot_wider()` a longer-format data
```{r akl-pivot, highlight.output = 1:2}
```
]

---

.left-column[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="60%">](http://tidyr.tidyverse.org)]
### - calibrate
### - pivot
]
.right-column[
<br>
.center[![](img/tidyr-longer-wider.gif)]
]

---

.left-column[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="60%">](http://tidyr.tidyverse.org)]
### - calibrate
### - pivot
### - rename
]
.right-column[
* `rename_with()` renames columns using a function
```{r akl-rename}
```
]

???

tedious to type

---

.left-column[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="60%">](http://tidyr.tidyverse.org)]
### - calibrate
### - pivot
### - rename
### - complete
]
.right-column[
* `complete()` data with missing combinations of data
```{r akl-complete, highlight.output = 1}
```
]

---

.left-column[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="60%">](http://tidyr.tidyverse.org)]
### - calibrate
### - pivot
### - rename
### - complete
]
.right-column[
* implicit missing records before
```{r akl-missing}
```
]

???

implicit bf, explicit after `complete()`

---

.left-column[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="60%">](http://tidyr.tidyverse.org)]
### - calibrate
### - pivot
### - rename
### - complete
### - wrangle
]
.right-column[
```{r akl-rainfall}
```
]

---

`r countdown::countdown(minutes = 0, seconds = 30, font_size = "48px", bottom = 0, left = 0)`

.pull-left[
## Your turn

> 1. Why do I fill `0` to missing `prcp`, instead of leaving `NA` as is?
> 2. Why do I calculate monthly cumulative precipitations?
]

.pull-right[
```{r akl-rainfall-na}
```
]

???

Now you need to think why I make this decision not that decision?

---

.left-column[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="60%">](http://tidyr.tidyverse.org)]
### - calibrate
### - pivot
### - rename
### - complete
### - wrangle
### - visualise
]
.right-column[
## Area plot for monthly cumulative rainfall
.panelset[

.panel[.panel-name[Plot]


```{r akl-rainfall-plot, echo = FALSE, fig.width = 7, fig.height = 3.5}
```

.panel[.panel-name[Code]

```{r ref.label = "akl-rainfall-plot", eval = FALSE}
```
]
]
]
]

???

* felt super try in 2020 summer
* Rainfall in last summer lower than the year before

---

.left-column[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="60%">](http://tidyr.tidyverse.org)]
### - calibrate
### - pivot
### - rename
### - complete
### - wrangle
]
.right-column[
```{r akl-monthly}
```
]

---

.left-column[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="60%">](http://tidyr.tidyverse.org)]
### - calibrate
### - pivot
### - rename
### - complete
### - wrangle
### - visualise
]
.right-column[
## Ribbon plot for monthly temperatures
.panelset[

.panel[.panel-name[Plot]


```{r akl-monthly-plot, echo = FALSE, fig.width = 7, fig.height = 3.5}
```

.panel[.panel-name[Code]

```{r ref.label = "akl-monthly-plot", eval = FALSE}
```
]
]
]
]

???

Why ribbon is missing in the end?

---

.left-column[
.center[[<img src="https://raw.githubusercontent.com/rstudio/hex-stickers/master/PNG/tidyr.png" width="60%">](http://tidyr.tidyverse.org)]
### - calibrate
### - pivot
### - rename
### - complete
### - wrangle
### - visualise
]
.right-column[
## Heatmap for monthly average temperatures
.panelset[

.panel[.panel-name[Plot]


```{r akl-heatmap, echo = FALSE, fig.width = 7, fig.height = 3.5}
```

.panel[.panel-name[Code]

```{r ref.label = "akl-heatmap", eval = FALSE}
```
]
]
]
]

---

## Reading

.pull-left[
.center[[<img src="https://d33wubrfki0l68.cloudfront.net/b88ef926a004b0fce72b2526b0b5c4413666a4cb/24a30/cover.png" height="520px">](https://r4ds.had.co.nz)]
]
.pull-right[
* [Tidy data](https://r4ds.had.co.nz/tidy-data.html)
* [{tidyr} cheatsheet](https://github.com/rstudio/cheatsheets/blob/master/data-import.pdf)
]
