Posts | Waseem Ashfaq

Flattening the leaRning curve: Teaching R online during COVID-19

Mon, 19 Oct 2020 00:00:00 +0000

At this point, you’re probably familiar with the phrase “flatten the curve” — remember back in March when we were all talking about that? It was part of a critical public health messaging campaign aimed at slowing the spread of COVID-19 to prevent hospitals from becoming overwhelmed beyond their capacity. Based on how things are going, we really should still be talking about that. So allow me to unabashedly steal this catchphrase as a flawed metaphor for teaching and learning R during the era of this pandemic…

It’s no secret that R has a significant leaRning curve — and, while it seems that more people than ever are now wanting to learn R, almost everyone is having to do it in an online format of some kind. For those who are used to a traditional in-person classroom environment, having a first introduction to R in an online format might make the leaRning curve even steeper (as I said, a flawed metaphor – the concept of a “steep learning curve” is actually the opposite of what most people think, but we’ll run with it anyway). After recently teaching an introductory R Bootcamp online, I’ve spent a lot of time thinking about strategies to lower the learning threshold and make it easier to conquer any potential fear of learning R – to “flatten the leaRning curve”, as it were.

For anyone who might be new to teaching R, teaching online, or both, the goal of this post is to share some valuable lessons I learned and resources I used during this online teaching experience. I hope it may be of some benefit for your own teaching.

Background

Back in Fall 2018, the Department of Psychology at the University of Oregon started to offer a grad-student-led introductory R Bootcamp the week before the start of the new academic year. The bootcamp consists of three 3-4 hour sessions that take place over the course of a week. The idea behind the bootcamp is to give a first exposure to R to incoming students who have little to no background in it before being launched headfirst into using R to analyze data for their first year project and to complete labs and homework assignments for PSY611, the first in a series of graduate statistics courses taught entirely in R.

In addition, as I’ve written about previously, current grad students, along with other members of the psychology department, have expressed a keen interest in developing a stronger foundation in skills such as basic programming, data wrangling, and data visualization. So the bootcamp also serves as an opportunity for existing R users to get a refresher and deeper dive into those topics for those who want it.

After spending lot of time thinking about how to improve the data science learning experience for my fellow grad students and having recently become an RStudio Certified Instructor in the tidyverse, it was my great privilege to be able to put all of this into action by leading the 3rd Annual UO Psych Bootcamp this year. 🥳

The challenge

However, I found myself facing a rather significant challenge. The R Bootcamp, like many intro R workshops, usually occurs in-person in a large room with a giant projector screen, coffee and snacks, and, most importantly, the ability to run over and look over someone’s shoulder at their computer screen when they need help. Because of COVID-19, I needed to design and deliver a bootcamp for a large group of beginners (many using R for the very first time) entirely over a webcam.

Moreover, I assumed that student motivation would be at an all-time low and feelings of anxiety and uncertainty would be at an all-time high – especially for those about to start their graduate school journey entirely online, very much contrary to plan. There were a lot of horrible things going on in the world in September 2020, which marked 6 months into the COVID-19 pandemic in the U.S. – people’s loved ones were getting sick, the killing of innocent Black lives continued, a bitter presidential race reached full swing – not to mention devastating wildfires that raged all along the west coast. (Of course, while I wish I could actually write about these events in the past tense, all of these problems continue).

So, how do you get people excited about learning R amidst all of that? Needless to say, this felt a little daunting 😨. Fortunately, though, there are people out there who are experts in this sort of thing who have graciously shared their resources and advice, and this helped me immensely to run a successful bootcamp.

Who attended?

52 people registered for the bootcamp. While the majority of those who attended were incoming first-year psychology grad students, we also had a mix of more advanced grad students, undergrads, research assistants, lab managers, staff data analysts, and faculty members from a variety of departments/centers across the University, including the Departments of Psychology, Linguistics, Economics and the Center for Translational Neuroscience.

# import bootcamp registration data 
registration <- read_csv("uopsych_bootcamp_registration_deidentified.csv")

# clean data
plot_data <- registration %>% 
  janitor::clean_names() %>% 
  select(position = what_is_your_position_at_uo,
         experience = do_you_have_any_experience_in_r_no_experience_is_required) %>%
  mutate(position = factor(case_when(position %in% c("Undergraduate", "Undergraduate student") ~ "Undergrad Student", 
                                     position %in% c("RA", "Lab Manager", "Data Analyst") ~ "Research Staff",
                              TRUE ~ as.character(position))),
         experience = factor(experience, levels = c(
           "I've never heard of it",
           "I've heard of it but have never used it",
           "I've opened it before",
           "I've used it to analyze data",
           "I use it often to analyze data")))

# plot positions at UO
theme_set(theme_minimal(base_size = 15)) 
theme_update(legend.position = "none", 
          plot.title.position = "plot",
          panel.grid.minor.y = element_blank())

plot_data %>% 
  count(position) %>% 
  ggplot(aes(fct_reorder(position, n), n, fill = position)) + 
  geom_col() +
  coord_flip() + 
  scale_fill_manual(values = get_pal("Kotare")) + 
  labs(x = "", y = "Count", title = "Participants' roles at University of Oregon")

# plot prior experience 
plot_data %>% 
  count(experience) %>% 
  ggplot(aes(experience, n, fill = experience)) + 
  geom_col() +
  coord_flip() + 
  scale_fill_manual(values = get_pal("Kaka"))  + 
  labs(x = "", y = "Count", title = "Participants' prior experience with R")

Logistics

Zoom + Slack = 👌

We ran synchronous sessions live over Zoom. However, rather than use the Zoom chat for communicating with each other and asking questions, we used Slack. A few reasons for this:

this way people could continue to ask questions outside of the live sessions and a permanent and searchable record of questions and answers would be available for posterity
The Zoom chat can get easily cluttered and doesn’t easily allow you to make threads, which keep things more organized
Slack has better emojis 🤓.

I was fortunate to have the very knowledgeable and talented Cameron Kay and Cianna Bedford-Petersen as teaching assistants. Cam and Cianna remained on deck throughout the sessions to answer questions on Slack while we went through the material. We also decided that if a student ran into technical issues or felt completely lost, then one of the TA’s would pull them into a breakout room on Zoom to offer one-on-one help.

RStudio Cloud

For “Your Turn” exercises, an idea I stole from Alison Hill’s excellent rstudio::conf(2020) workshop, we used RStudio Cloud, a hosted version of the RStudio IDE available in the web browser. We did this to avoid issues with students installing RStudio on their local machines, as troubleshooting technical issues over Zoom in a large group is not something we felt prepared for. Using RStudio Cloud also allowed me to configure specific IDE settings and pre-load specific packages to allow students to initially avoid technical issues. Lastly, sharing a workspace on RStudio Cloud allowed each student to save their own copy of the projects containing the Your Turn exercises, which in turn made it possible for myself or the TA’s to “peek” into a student’s exercises and help them figure out where their errors were coming from. Overall it went really smoothly, and everyone who filled out the feedback survey said they had no issues using it. I highly recommend using this for your teaching, even if it’s in-person.

Blogdown site

To make the bootcamp materials more accessible and shareable, I decided to package them into a website, which I created using the blogdown package (see more on this below). Organizing your teaching materials into a website has three clear advantages in my mind:

A website makes your materials easier to access and more shareable – you send someone a single link they can access on their phone or computer, no need to download anything
A website is highly sustainable in the sense that it’s easy for students to bookmark and refer back to over and over, and if you push changes to Github with continuous deployment (e.g. via Netlify), students will always have access to the most updated materials. This avoids having to introduce Github and version control, which is a whole other beast to teach to beginners.
Websites are pretty and highly customizble ✨

🔗 uopsych-r-bootcamp-2020.netlify.app

Tools I learned

`xaringanExtra`

📦 pkg.garrickadenbuie.com/xaringanExtra

Garrick Aden-Buie

{xaringanExtra} extends the functionality of the already amazing {xaringan} package by Yihuie Xie, and it has become one of my all-time favorite packages. Here are the features I use most often and highly recommend.

📖 Share again

📖 Panelset

📖 Tile view

📖 Extra styles

In case it still isn’t obvious enough how much I love this package…

Ok, {xaringanExtra} continues to blow my freakin' mind 🤯 Here's a quick demo that shows tile view and embed_xaringan() in the context of a #ggplot2 flipbook (h/t @EvaMaeRey). @grrrck, I owe you an absurd amount of beers 🍻 Thank you for all you do for the #rstats community 🙌 pic.twitter.com/ucMCbOvopW
— Brendan Cullen (@_bcullen) September 28, 2020

Jump back up to see more tools ⤴️

`flair`

📦 r-for-educators.github.io/flair

Kelly Bodwin

{flair} allows you to spice up your code with color-coding and dynamic syntax highlighting. One of the most useful applications is to match colors to concepts – e.g. giving each argument of a function a unique color. I feel like I’ve only scratched the surface with {flair} and can’t wait to use it more in my teaching going forward.

Below is a demonstration of color-coding with flair to introduce dplyr::filter() and its arguments.

You can read more about other educational use cases for flair in this great post on the RStudio Education Blog.

Jump back up to see more tools ⤴️

`flipbookr`

📦 evamaerey.github.io/flipbooks/about

Gina Reynolds

{flipbookr} is an amazing teaching tool – it allows you to incrementally build up a plot line by line, which capitalizes on the “blink comparator” method. This means that students are able to easily map a specific change in code to a specific visual change in the corresponding plot output. Flipbooks are also great for demonstrating incremental changes in a data wrangling pipeline, and they’re relatively easy to make!

Video

Jump back up to see more tools ⤴️

`countdown`

📦 pkg.garrickadenbuie.com/countdown

Garrick Aden-Buie

{coundown} is another great gem from Garrick and allows you to put timers directly on your slides. This is really useful for timed “Your Turn” exercises or to make sure you stay on track with Q & A sessions or breaks.

Video

Jump back up to see more tools ⤴️

The importance of encouragement & positivity

The leaRning curve is no secret. I hadn’t touched R before starting grad school in 2017, but I had heard whispers here and there that learning R is really difficult. But I think there’s an element of self-fulfilling prophecy here. If you expect it to be impossible, then it will feel impossible. I more or less started my journey with that expectation.

Things changed for me when I took my first course with Daniel Anderson. Daniel has developed a fantastic 5-course Data Science Specialization in Educational Leadership in the College of Education at the University of Oregon, taught entirely in R, which ranges from the very basics of R to advanced machine learning with tidymodels (see more info here). Daniel’s teaching completely changed the way I approached using and learning R, and it was largely on his recommendation that I joined #rstats Twitter (which was a game-changer). More importantly, Daniel modeled the kind of teacher I wanted to be – clear, thorough, thoughtful, and above all else, kind and inclusive of everyone.

It was very much with Daniel’s model in mind that I found inspiration for making learning R feel more inviting and less daunting. Here are a few things that came to mind that can be abstracted to some general advice…

Anticipate the challenge

Recognize the fact that learning R is hard. Don’t pretend it’s a walk in the park – it isn’t. But at the same time, emphasize that the investment of time it takes is worth it! And be sure to remind people of that more than once 😉.

Normalize errors & mistakes

The first time someone uses R, error messages are one of the first things that they’ll see. As we all know, this feels frustrating, especially as a beginner. So it makes sense to acknowledge that up front and discuss what error messages are and why they occur. As error messages go hand-in-hand with debugging, it’s also a good idea to at least introduce that concept as well, without necessarily going into great depth since that’s a whole other topic of it’s own. As debugging is commonly considered part of “what they forgot to teach you about R”, I’m sure I’m not alone in wishing that I had been introduced to debugging much earlier on in my R learning experience.

While learning a systematic approach to debugging is important down the line, it’s more important at first for beginners to understand that seeing lots of error messages at first is not a bad thing. It’s worth spending some effort to convince your students that error messages should not be cause for discouragement or giving up. Simply put, error messages are part and parcel of programming. Sometimes they take a matter of seconds to fix, sometimes a matter of weeks, but everyone who uses R will regularly encounter error messages, no matter your level of experience. And when it really comes down to it, every error message is an opportunity to learn something!

Taking this one step further, it’s a good idea to deliberately incorporate making mistakes into your own teaching and use error messages as a teachable moment. You can even use unintentional mistakes as teachable moments – demonstrate for your students how you fix the mistake and invite them to give their thoughts and feedback. This not only makes your presentation less rehearsed, but it provides a really valuable “real-world” context for learning.

This idea is referred to as “positive error framing” – and if you want to learn more about this topic in depth I recommend you check out this section from the recent article “Ten quick tips for teaching with participatory live coding” (really you should read the whole article – it’s packed full of great advice).

Make your materials friendly and inviting!

If your slides read like one long piece of dense help documentation, people are not going to feel inclined to engage with it. How do you make your materials more inviting and approachable?

One of the most powerful tools out there, in my opinion, is the incredible artwork by Allison Horst, RStudio’s first artist-in-residence (see more above). The bigger picture behind the artwork, in Allison’s words:

“All of it is towards a goal of trying to make R, and data science in general, more approachable, welcoming, and engaging to diverse learners”.

Cute gifs never hurt either…

If you're teaching #rstats with #palmerpenguins at the moment (which you should be because it's a brilliant dataset 🐧), consider sneaking in a lil' motivational penguin, both for yourself and your students. everything is terrible and we all need small joys now more than ever pic.twitter.com/OHeJWIsqFp
— Brendan Cullen (@_bcullen) September 24, 2020

Feedback

I was really happy to see that people seemed to appreciate the positive learning atmosphere and, especially because this was only an introductory workshop, that this made them excited to learn more:

✍️ “I really appreciated the warm welcome/atmosphere the instructors set up- learning a new programming language can be frustrating, especially for a beginner, so I was really appreciative of the continuous encouragement throughout the bootcamp.”

✍️ “I really enjoyed how supportive and encouraging the facilitators were. I understood learning R was no easy endeavor but hearing about their experiences was encouraging for me. Also, I greatly appreciate all the amazing resources they have compiled for future use. I definitely plan to continue looking into it!”

✍️ “I appreciated the facilitators’ organization, support, encouragement, and instructional clarity. Thank you for providing this very warm and accessible introduction for an absolute first-time novice! It’s encouraged me to take more steps into learning R.”

✍️ “I am now in my third year of struggling through R and this was the first presentation of material that really clicked for me. I frequently refer to the dplyr and ggplot presentations in my work and will continue to work my way through these exercises until I am truly fluent in these basics. THANK YOU”

So we did we flatten the leaRning curve? Only time will tell. But the most gratifying feedback, in my opinion, is that people who attended felt inspired to keep learning and continue riding the R roller coaster. That’s really the best outcome I could have hoped for!

A final thought

In short, no matter how long someone has been using R, it never hurts to remind them (yourself included) that a little time and effort can go a long way.

Acknowledgements

I owe a huge debt of gratitude to all the people who were instrumental in my initial experience of learning R: Daniel Anderson, Dani Cosme, Krista DeStasio, Cory Costello, Jessica Kosie, and Sara Weston, to name only a few.

Thanks again to Cameron Kay and Cianna Bedford-Petersen for being such awesome TA’s for the R Bootcamp.

I also wouldn’t know half of what I know about learning and teaching R if it weren’t for the RStudio Education team – in particular, Alison Hill, Greg Wilson, Desirée De Leon, Mine Çetinkaya-Rundel, and Garrett Grolemund.

Artwork included in this post is by Allison Horst.

Session Info

## ─ Session info ───────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.2 (2019-12-12)
##  os       macOS Catalina 10.15.6      
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/Los_Angeles         
##  date     2020-11-01                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────
##  package       * version    date       lib
##  assertthat      0.2.1      2019-03-21 [1]
##  backports       1.1.10     2020-09-15 [1]
##  blogdown        0.17       2019-11-13 [1]
##  bookdown        0.21       2020-10-13 [1]
##  broom           0.5.5      2020-02-29 [1]
##  cellranger      1.1.0      2016-07-27 [1]
##  cli             2.1.0      2020-10-12 [1]
##  colorspace      1.4-1      2019-03-18 [1]
##  crayon          1.3.4      2017-09-16 [1]
##  DBI             1.1.0      2019-12-15 [1]
##  dbplyr          1.4.2      2019-06-17 [1]
##  digest          0.6.27     2020-10-24 [1]
##  dplyr         * 1.0.2      2020-08-18 [1]
##  ellipsis        0.3.1      2020-05-15 [1]
##  emo             0.0.0.9000 2020-08-26 [1]
##  evaluate        0.14       2019-05-28 [1]
##  fansi           0.4.1      2020-01-08 [1]
##  farver          2.0.3      2020-01-16 [1]
##  forcats       * 0.4.0      2019-02-17 [1]
##  fs              1.3.1      2019-05-06 [1]
##  generics        0.0.2      2018-11-29 [1]
##  ggplot2       * 3.3.2      2020-06-19 [1]
##  glue            1.4.2      2020-08-27 [1]
##  gtable          0.3.0      2019-03-25 [1]
##  haven           2.2.0      2019-11-08 [1]
##  highr           0.8        2019-03-20 [1]
##  hms             0.5.3      2020-01-08 [1]
##  htmltools       0.5.0      2020-06-16 [1]
##  httr            1.4.2      2020-07-20 [1]
##  janitor         1.2.1      2020-01-22 [1]
##  jpeg            0.1-8.1    2019-10-24 [1]
##  jsonlite        1.7.1      2020-09-07 [1]
##  knitr           1.30       2020-09-22 [1]
##  labeling        0.4.2      2020-10-20 [1]
##  lattice         0.20-38    2018-11-04 [2]
##  lifecycle       0.2.0      2020-03-06 [1]
##  lubridate       1.7.9      2020-06-08 [1]
##  magrittr        1.5        2014-11-22 [1]
##  Manu          * 0.0.1      2020-10-20 [1]
##  modelr          0.1.6      2020-02-22 [1]
##  munsell         0.5.0      2018-06-12 [1]
##  nlme            3.1-142    2019-11-07 [2]
##  pillar          1.4.6      2020-07-10 [1]
##  pkgconfig       2.0.3      2019-09-22 [1]
##  png             0.1-7      2013-12-03 [1]
##  purrr         * 0.3.4      2020-04-17 [1]
##  R6              2.4.1      2019-11-12 [1]
##  Rcpp            1.0.5      2020-07-06 [1]
##  readr         * 1.3.1      2018-12-21 [1]
##  readxl          1.3.1      2019-03-13 [1]
##  reprex          0.3.0      2019-05-16 [1]
##  rlang           0.4.8      2020-10-08 [1]
##  rmarkdown       2.5        2020-10-21 [1]
##  rstudioapi      0.11       2020-02-07 [1]
##  rvest           0.3.5      2019-11-08 [1]
##  scales          1.1.1      2020-05-11 [1]
##  sessioninfo     1.1.1      2018-11-05 [1]
##  snakecase       0.11.0     2019-05-25 [1]
##  stringi         1.5.3      2020-09-09 [1]
##  stringr       * 1.4.0      2019-02-10 [1]
##  tibble        * 3.0.4      2020-10-12 [1]
##  tidyr         * 1.1.0      2020-05-20 [1]
##  tidyselect      1.1.0      2020-05-11 [1]
##  tidyverse     * 1.3.0      2019-11-21 [1]
##  vctrs           0.3.4      2020-08-29 [1]
##  withr           2.3.0      2020-09-22 [1]
##  xaringanExtra * 0.2.1      2020-08-23 [1]
##  xfun            0.18       2020-09-29 [1]
##  xml2            1.2.2      2019-08-09 [1]
##  yaml            2.2.1      2020-02-01 [1]
##  source                                  
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.2)                          
##  Github (hadley/emo@3f03b11)             
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.0)                          
##  Github (G-Thomson/Manu@c5aa932)         
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.2)                          
##  Github (gadenbuie/xaringanExtra@f85b70a)
##  CRAN (R 3.6.2)                          
##  CRAN (R 3.6.0)                          
##  CRAN (R 3.6.0)                          
## 
## [1] /Users/bcullen/Library/R/3.6/library
## [2] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

Reflections on RStudio Instructor Training

Thu, 03 Sep 2020 00:00:00 +0000

A while back I wrote about how we need more data science training as grad students in psychology and that one of the best ways for us to get this training is to learn from each other. This is just one of many reasons why I’m so humbled and excited to have recently become an RStudio Certified Instructor in the tidyverse.

I’m looking forward to start implementing and sharing what I learned with my fellow grad students, especially at the end of this month when I will be leading an introductory R workshop for the new cohort of first year PhD and master’s students in my department.

As a small way of paying it forward, I wanted to offer a reflection on what I think makes this training so unique and worthwhile along with a summary of what’s involved and some resources for those who might be interested in knowing more.

The what

The best place the start is to read about the RStudio Instructor Training and Certification Program here. The RStudio Education Blog also has lots of helpful posts about the program.

The process boils down to three steps.

Training Course

The first step is to sign up for the training course, which will likely be held over Zoom and chunked into 3-4 hour segments across 2-3 days. You can access the training materials here.

The training course is an interactive introduction to applying evidence-based teaching methods to data science education, such as learner personas, concept maps, and formative assessment (i.e. short, diagnostic questions or exercises to figure out if learners are forming accurate mental models). The course also covers how to design teaching materials with reverse instructional design that takes into account cognitive load, multiple learning strategies, and issues of inclusivity and student motivation in the classroom.

Technical Exam

The technical exam assesses your proficiency in whatever topic for which you are attempting to become a certified instructor. Currently there are options to become a certified instructor in tidyverse and shiny, and each has its own accompanying technical exam.

I took the tidyverse exam, which, broadly speaking, consists of a series of live coding challenges related to using core tidyverse packages for data cleaning and wrangling, data visualization, string manipulation, functional programming, basic statistical modeling, and creating reproducible documents with R Markdown.

As many others have suggested, a great way to prepare for this exam is to work through the exercises R for Data Science, particularly for topics that feel rusty to you, and review the community-contributed solutions.

I highly recommend going through these sample exams from the RStudio Education Blog start to finish to get a sense of what you might need to review.

Teaching Exam

The first certification exam assesses pedagogical skills related to teaching data science with R and requires giving a 15-minute demonstration lesson on a topic of your choice followed by a series of applied questions, which will likely involve creating formative assessments on unseen material (e.g. multiple choice questions and fill-in-the-blank coding exercises), developing concept maps on data science topics and giving feedback on example teaching based on pedagogical theory.

There are also sample teaching exams available.

Demonstration Lesson: Example

Column-wise operations with dplyr: Old and New

If you’d like to see an example of a demonstration lesson, below are the materials I created for this portion of the teaching exam. I used penguins from the {palmerpenguins} package as an example data set. (why? because penguins 🐧 🐧 🐧).

You can find all of the materials for this lesson and the accompanying code on Github. Feel free to share, adapt and re-use for your own teaching.

Slides

I made heavy use of Yihui Xie’s {xaringan} 📦, Garrick Aden-Buie’s {xaringanExtra} 📦, and Kelly Bodwin’s {flair} 📦, along with Allison Horst’s unbeatable artwork. For an excellent {xaringan} tutorial, I recommend you check out these slides, from the R Markdown whisperer herself, Alison Hill. Note: you absolutely do not have to use {xaringan} to make your slides, and if your lesson includes more images than code, another method for delivering your slides might be better.

Full slides available here.

❖ ❖ ❖

Concept Map

For other community-contributed data science concept maps you can use in your teaching and/or lesson prep, see here.

❖ ❖ ❖

Learner Persona

For a list of other example learner personas, see here.

❖ ❖ ❖

Formative Assessment

I created these interactive exercises using the learnr package, which I highly recommend you check out. It’s quite powerful and versatile.

Here’s a quick look.

The why

Ok, this might all seem like quite a bit of time and effort. Why go to the trouble of doing this training? In a word, Greg Wilson.

Greg, who co-founded the Software Carpentry, has over 35 years of experience in education in data science and software engineering, and it shows. He is now part of the RStudio Education team, where he runs the the instructor training and certification program. One of the reasons this program stands out is that it benefits from Greg’s unique expertise and careful curation of decades of research on evidence-based teaching methods that he has translated into clear and actionable advice. I can guarantee that you will learn a LOT from him.

Greg’s most important advice for teaching, in my opinion:

“Be kind: all else is details.”

Now, here are some other reasons why you should do this training…

Surge in online teaching

Interest in data science education seems to be ever-increasing. The fact that COVID-19 has forced most education to go online might actually present an opportunity to meet this demand in a more scalable and (hopefully more accessible way that doesn’t incur the traditional limitations of travel costs or room capacity. Of course, online education comes with a host of inherent challenges. The training course includes a whole section on this. I also recommend you check out this RStudio webinar and accompanying blog post along with answers to some frequently asked questions about teaching online.

As online data science education is becoming increasingly the norm, it seems natural to assume that there will be a need for more certified instructors to meet the growing demand.

❖ ❖ ❖

Teaching resources galore

Another great reason to become a certified instructor is that, as a data science educator, you have a huge and ever-increasing bank of resources at your disposal. What’s more, as a certified instructor, you are eligible for free licenses to RStudio Pro products and a significant discount for RStudio Cloud. Here are just some of the great teaching tools from RStudio and the #rstats community.

The RStudio Education Blog is a 💎 TREASURE TROVE 💎 of resources. Add it to your bookmarks immediately.

Teaching with RStudio Cloud

webinar and accompanying blog post, courtesy of Mine Çetinkaya-Rundel

Interactive lessons with learnr

tutorial and demonstration, courtesy of Allison Horst
lessons learned from using learnr to teach high schoolers, courtesy of Ezgi Karaesmen
a deep dive into learnr, courtesy of Nischal Shrestha
{gradethis} package for easily grading learnr exercises

Openly licensed teaching materials

a thorough summary of recent online courses/workshops, courtesy of Alison Hill
Data Science in a Box, courtesy of Mine Çetinkaya-Rundel
STAT 545, courtesy of Jenny Bryan
Many others, courtesy of the RStudio Education Team

rstudio4edu

a book-in-progress, courtesy of Desirée De Leon and Alison Hill. This is in a category of its own because it’s an absolute gold mine.

❖ ❖ ❖

Tried and tested

The instructor training program started back in February 2019 and as of August 2020 there are almost 150 certified tidyverse instructors and 20 shiny instructors. This means that the program has gone through multiple iterations and has made data-driven improvements based on feedback from participants – especially in the realm of supporting online teaching in the aftermath of COVID-19. So you can rest assured that, while it is still a relatively new program, all the kinks have been worked out.

Plus, I’m sure that the content and structure of the training will continue to adapt to the needs and priorities of the community, and you might even be lucky enough to catch a special guest presentation. For example…

Enjoyed sharing some tips today on “Teaching in Production” for our [@RStudioEDU] certified-instructors-to-be (another [@thomas_mock] distill success story!)

Thx to [@gvwilson] for the invite, [@allison_horst] for #rmarkdown 🦔(❣️), and [@dcossyle] for all her 🎨.https://t.co/sngAwlcSqN

— Alison Presmanes Hill ([@apreshill]) August 20, 2020

❖ ❖ ❖

A focus on inclusivity

The focus of this training is not technical competency – it’s how be an effective teacher. One of the most critical components of teaching effectively is to be inclusive of all learners, regardless of race, religion, sexual orientation, gender identity, disability, etc.

The #rstats learning community is known for being welcoming and inclusive, so it’s no surprise that the training course emphasizes these values as well. What I appreciate most about this aspect of the training is that it will challenge you to think about questions and hypothetical scenarios to which there are no easy answers.

However, it is extremely important to be pushed out of your comfort zone to consciously and proactively reflect on how you will confront issues such as systemic racism and institutionalized violence against BIPOC communities, sexism and a deeply ingrained culture of sexual harassment. These issues will inevitably arise in one form or another in your classroom or teaching setting, and it’s absolutely necessary that we confront these challenges now more than ever. Check out this slide and this talk on effective allyship when you get a chance.

For further reading on inclusivity and social justice in data science education, I recommend you read this post by Nicole Thompson Gonzalez and this one by Yim Register. Also check out the amazing work that JooYoung Seo, the first blind RStudio Certified Instructor, has been doing to make data science tools more accessible.

Another exciting feature is that the training materials are now available in Spanish, courtesy of Laura Acion, and hopefully other languages soon, as interest in the training seems to be growing around the world. A similar ongoing project to check out is glosario, an open source glossary of data science terms translated in multiple languages that can be used for teaching (read more here).

❖ ❖ ❖

Community of practice

To extend the idea of including everyone who wants to learn data science, we must be active in building teaching communities that extend beyond just the walls of academic institutions.

RStudio’s mission is to

“equip everyone, regardless of means, to participate in a global economy that rewards data literacy.”

A more concrete goal, put forth by Carl Howe, Director of Education at RStudio, is to train the next million R users. In becoming an RStudio certified instructor, you can better position yourself to actively participate in reaching this goal. But it’s worth reflecting on the fact that teaching and learning doesn’t happen in a vacuum – this is where the idea of community comes in. In my opinion, the fact that R users around the world already have a strong sense of community will make it that much easier to welcome new learners into the fold and make it more likely that they themselves will start to train others one day.

Read more about building a community of practice here.

P.S. If you need yet another reason to do this training, you get a fancy certificate at the end. ✨

Get in touch

Please feel free to reach out if you are thinking of participating in the training yourself and want to hear more from someone who’s gone through it recently. I would be glad to chat any time!

Tidymodels: Decision Tree Learning in R

Tue, 02 Jun 2020 00:00:00 +0000

RStudio has recently released a cohesive suite of packages for modelling and machine learning, called {tidymodels}. The successor to Max Kuhn’s {caret} package, {tidymodels} allows for a tidy approach to your data from start to finish. We’re going to walk through the basics for getting off the ground with {tidymodels} and demonstrate its application to three different tree-based methods for predicting student test scores. For further information about the package, you can visit https://www.tidymodels.org/.

Setup

Load both the {tidyverse} and {tidymodels} packages into your environment. We’ll also load in the {skimr} package to help us with some descriptives for our data and a host of other packages that will be required to run our machine learning models.

library(tidymodels) 
library(tidyverse) # manipulating data
library(skimr) # data visualization
library(baguette) # bagged trees
library(future) # parallel processing & decrease computation time
library(xgboost) # boosted trees

Import the data

We use simulated data which approximates reading and math scores for ~189,000 3rd-8th grade students in Oregon public schools see this Kaggle page for details. For the purpose of demonstration, we’ll be sampling 1% of the data with sample_frac() to keep computer processing time manageable. All school IDs in the data are real, so we can use that information to link the data with other sources. Specifically, we’re also going to pull in some data on student enrollment in free and reduced lunch from the National Center for Education Statistics and some ethnicity data from the Oregon Department of Education.

set.seed(100)

# import data and perform initial cleaning
# initial cleaning steps include: 
# *recode NA's for lang_cd and ayp_lep to more meaningful values
# *remove vars with entirely missing data
# Note: the data is called 'train.csv', but we will actually further split this into its own training and testing data

dat <- read_csv(here::here("static", "data", "train.csv")) %>% 
  select(-classification) %>% # remove this variable because it's redundant with `score`
  mutate(lang_cd = ifelse(is.na(lang_cd), "E", lang_cd), 
         ayp_lep = ifelse(is.na(ayp_lep), "G", ayp_lep)) %>% 
  sample_frac(.01) %>% # sample 1% of the data to reduce run time
  janitor::remove_empty(c("rows", "cols")) %>% 
  drop_na() %>% 
  select_if(~length(unique(.x)) > 1)

# import fall membership report ethcnicity data and do some basic cleaning and renaming
sheets <- readxl::excel_sheets(here::here("static", "data", "fallmembershipreport_20192020.xlsx"))

ode_schools <- readxl::read_xlsx(here::here("static", "data", "fallmembershipreport_20192020.xlsx"),
                                 sheet = sheets[4])

ethnicities <- ode_schools %>%
  select(attnd_schl_inst_id = `Attending School ID`,
         attnd_dist_inst_id = `Attending District Institution ID`,
         sch_name = `School Name`,
         contains("%")) %>%
  janitor::clean_names()

names(ethnicities) <- gsub("x2019_20_percent", "p", names(ethnicities))

# join ethnicity data with original dataset
dat <- left_join(dat, ethnicities)

# import and tidy free and reduced lunch data 
frl <- rio::import("https://nces.ed.gov/ccd/Data/zip/ccd_sch_033_1718_l_1a_083118.zip",
              setclass = "tbl_df")  %>% 
  janitor::clean_names()  %>% 
  filter(st == "OR")  %>%
  select(ncessch, lunch_program, student_count)  %>% 
  mutate(student_count = replace_na(student_count, 0))  %>% 
  pivot_wider(names_from = lunch_program,
              values_from = student_count)  %>% 
  janitor::clean_names()  %>% 
  mutate(ncessch = as.double(ncessch))

# import student counts for each school across grades
stu_counts <- rio::import("https://github.com/datalorax/ach-gap-variability/raw/master/data/achievement-gaps-geocoded.csv", setclass = "tbl_df")  %>% 
  filter(state == "OR" & year == 1718)  %>% 
  count(ncessch, wt = n)  %>% 
  mutate(ncessch = as.double(ncessch))

# join frl and stu_counts data
frl <- left_join(frl, stu_counts)

# add frl data to train data
dat <- left_join(dat, frl)

After loading in our three datasets, we’ll join them together to make one cohesive data set to use for modelling. After joining, the data contains both student-level variables (e.g. gender, ethnicity, enrollment in special education/talented and gifted programs, etc.) and district-level variables (e.g. school longitude and latitude, proportion of students who qualify for free and reduced-price lunch, etc.), all of which will be included for each 3 of our {tidymodels} tree-based examples.

For a more complete description of the variables, you can download the data dictionary here.

Explore the data

We’ll use the skim() function from {skimr} to take a closer look at our variables. Many numeric predictors are clearly non-normal (see histograms below), but this is no problem as tree-based methods are robust to non-normality.

dat %>% 
  select(-contains("id"), -ncessch, -missing, -not_applicable) %>%  # remove ID and irrelevant variables
  mutate(tst_dt = lubridate::as_date(lubridate::mdy_hms(tst_dt))) %>% # covert test date to date
  modify_if(is.character, as.factor) %>%  # convert character vars to factors
  skim() %>% 
  select(-starts_with("numeric.p")) # remove quartiles

Table 1: Data summary
Name	Piped data
Number of rows	1857
Number of columns	41
_______________________
Column type frequency:
Date	1
factor	25
numeric	15
________________________
Group variables	None

Variable type: Date

skim_variable	n_missing	complete_rate	min	max	median	n_unique
tst_dt	0	1	2018-03-16	2018-06-07	2018-05-18	47

Variable type: factor

skim_variable	n_missing	complete_rate	ordered	n_unique	top_counts
gndr	0	1	FALSE	2	M: 939, F: 918
ethnic_cd	0	1	FALSE	7	W: 1151, H: 458, M: 100, A: 79
tst_bnch	0	1	FALSE	6	G6: 343, 1B: 330, G4: 304, G7: 304
migrant_ed_fg	0	1	FALSE	2	N: 1793, Y: 64
ind_ed_fg	0	1	FALSE	2	N: 1842, Y: 15
sp_ed_fg	0	1	FALSE	2	N: 1614, Y: 243
tag_ed_fg	0	1	FALSE	2	N: 1759, Y: 98
econ_dsvntg	0	1	FALSE	2	Y: 1100, N: 757
ayp_lep	0	1	FALSE	10	G: 1471, F: 164, Y: 72, E: 58
stay_in_dist	0	1	FALSE	2	Y: 1811, N: 46
stay_in_schl	0	1	FALSE	2	Y: 1803, N: 54
dist_sped	0	1	FALSE	2	N: 1846, Y: 11
trgt_assist_fg	0	1	FALSE	3	N: 1773, Y: 83, y: 1
ayp_schl_partic	0	1	FALSE	2	Y: 1846, N: 11
ayp_dist_prfrm	0	1	FALSE	2	Y: 1803, N: 54
ayp_schl_prfrm	0	1	FALSE	2	Y: 1785, N: 72
rc_schl_partic	0	1	FALSE	2	Y: 1846, N: 11
rc_dist_prfrm	0	1	FALSE	2	Y: 1803, N: 54
rc_schl_prfrm	0	1	FALSE	2	Y: 1785, N: 72
lang_cd	0	1	FALSE	2	E: 1815, S: 42
tst_atmpt_fg	0	1	FALSE	2	Y: 1853, P: 4
grp_rpt_schl_partic	0	1	FALSE	2	Y: 1846, N: 11
grp_rpt_dist_prfrm	0	1	FALSE	2	Y: 1845, N: 12
grp_rpt_schl_prfrm	0	1	FALSE	2	Y: 1834, N: 23
sch_name	1	1	FALSE	699	Hig: 14, Jud: 14, Hou: 13, Fiv: 11

Variable type: numeric

skim_variable	n_missing	complete_rate	mean	sd	hist
enrl_grd	0	1	5.44	1.69	▇▃▅▃▃
score	0	1	2495.34	115.19	▁▁▂▇▁
lat	0	1	44.79	0.99	▂▁▂▅▇
lon	0	1	-122.51	1.16	▅▇▁▁▁
p_american_indian_alaska_native	1	1	0.01	0.06	▇▁▁▁▁
p_asian	1	1	0.04	0.07	▇▁▁▁▁
p_native_hawaiian_pacific_islander	1	1	0.01	0.01	▇▁▁▁▁
p_black_african_american	1	1	0.02	0.04	▇▁▁▁▁
p_hispanic_latino	1	1	0.25	0.18	▇▅▂▁▁
p_white	1	1	0.60	0.20	▁▃▅▇▅
p_multiracial	1	1	0.06	0.03	▇▆▁▁▁
free_lunch_qualified	0	1	231.23	147.55	▇▇▃▁▁
reduced_price_lunch_qualified	0	1	39.86	24.77	▆▇▃▁▁
no_category_codes	0	1	271.09	165.44	▆▇▃▁▁
n	0	1	816.07	536.55	▇▃▂▁▁

While most of our predictors are categorical, we can use {corrplot} to better visualize the relationships among the numeric variables.

dat %>% 
  select(-contains("id"), -ncessch, -missing, -not_applicable) %>% 
  select_if(is.numeric) %>% 
  select(score, everything()) %>% 
  cor(use = "complete.obs") %>% 
  corrplot::corrplot()

Split data and resample

The first step of our analysis is to split our data into two separate sets: a “training” set and a “testing” set. The training set is used to train a model and, if desired, to adjust (i.e., “tune”) the model’s hyperparameters before evaluating its final performance on our test data. By allowing us to test a model on a new sample, we assess “out of sample” accuracy (i.e., unseen data-—what all predictive models are interested in) and limit overfitting to the training set. We can do this efficiently with the initial_split() function. This comes from the {rsample} package, which is part of the {tidymodels} package that we already loaded. Defaults put 75% of the data in the training set and 25% in the test set, but this can be adjusted with the prop argument. Then, we’ll extract the training data from our split object and assign it a name.

To further prevent over-fitting, we’ll resample our data using vfold_cv(). This function outputs k-fold cross-validated versions of our training data, where k = the number of times we resample (unsure why v- is used instead of k- here). By using k = 10 data sets, we get a better estimate of the model’s out-of-sample accuracy. On top of decreasing bias from over-fitting, this is essential when tuning hyperparameters (though we plan to apply defaults and not tune here, for brevity). Though our use of 10-fold cross validation is both frequently used and effective, it should be noted that other methods (e.g., bootstrap resampling) or other k-values are sometimes used to accomplish the same goal.

# split the data
split <- initial_split(dat)

# extract the training data
train <- training(split)

# resample the data with 10-fold cross-validation (10-fold by default)
cv <- vfold_cv(train)

Pre-processing

Before we add in our data to the model, we’re going to set up an object that pre-processes our data. This is called a recipe. To create a recipe, you’ll first specify a formula for your model, indicating which variable is your outcome and which are your predictors. Using ~. here will indicate that we want to use all variables other than score as predictors. Then, we can specify a series of pre-processing steps for our data that directs our recipe to assign our variables a role or performs feature engineering steps. Pre-processing may be sound uncommon, but if you’ve ever used lm() (or several other R functions) you’ve done some of this by simply calling the function (e.g., automatic dummy-coding to handle categorical data). This is beneficial because it gives the analyst more control, despite adding complexity to the process.

A complete list of possible pre-processing steps can be found here: https://www.tidymodels.org/find/recipes/

rec <- recipe(score ~ ., train) %>% 
  step_mutate(tst_dt = as.numeric(lubridate::mdy_hms(tst_dt))) %>% # convert `test date` variable to a date 
  update_role(contains("id"), ncessch, new_role = "id vars") %>% # declare ID variables
  step_nzv(all_predictors(), freq_cut = 0, unique_cut = 0) %>% # remove variables with zero variances
  step_novel(all_nominal()) %>% # prepares test data to handle previously unseen factor levels 
  step_unknown(all_nominal()) %>% # categorizes missing categorical data (NA's) as `unknown`
  step_medianimpute(all_numeric(), -all_outcomes(), -has_role("id vars"))  %>% # replaces missing numeric observations with the median
  step_dummy(all_nominal(), -has_role("id vars")) # dummy codes categorical variables

Create a model

The last step before bringing in our data is to specify our model. This will call upon functions from the {parsnip} package, which standardizes language for specifying a multitude of statistical models. There are a few core elements that you will need to specify for each model

The type of model

This indicates what type of model you choose to fit, each of which will be a different function. We’ll be focusing on decision tree methods using bag_tree(), random_forest(), and boost_tree(). A full list of models can be found here https://www.tidymodels.org/find/parsnip/

The engine

set_engine() calls the package to support the model you specified above.

The mode

set_mode() indicates the type of prediction you’d like to use in your model, you’ll choose between regression and classification. Since we are looking to predict student scores, which is a continuous predictor, we’ll be choosing regression.

The arguments

set_args() allows you to set values for various parameters for your model, each model type will have a specific set of parameters that can be altered. For these parameters, you can either set a particular value or you can use the tune function to search for the optimal value of each parameter. Tuning requires a few extra steps, so we will leave the default arguments for clarity. For more information on tuning check out https://tune.tidymodels.org/.

Create a workflow

Up to this point we’ve been setting up a lot of individual elements and now it is time to combine them to create a cohesive framework, called a workflow, so we can run our desired models. First, we’ll use the workflow() command and then we’ll pulling the recipe and model we already created. The next section shows three examples of specifying models and creating a workflow for different decision tree methods.

Model Examples

Bagged trees

A bagged tree approach creates multiple subsets of data from the training set which are randomly chosen with replacement. Each subset of data is used to train a given decision tree. In the end, we have an ensemble of different models. The predictions from all the different trees are averaged together, giving us a stronger prediction than one tree could independently.

Specify model

set.seed(100)
mod_bag <- bag_tree() %>%
  set_mode("regression") %>%
  set_engine("rpart", times = 10) # 10 bootstrap resamples

Create workflow

wflow_bag <- workflow() %>% 
  add_recipe(rec) %>%
  add_model(mod_bag)

Fit the model

set.seed(100)
plan(multisession)

fit_bag <- fit_resamples(
  wflow_bag,
  cv,
  metrics = metric_set(rmse, rsq),
  control = control_resamples(verbose = TRUE,
                              save_pred = TRUE,
                              extract = function(x) extract_model(x)))

Visualize

The plot below shows the root nodes from a bagged tree made of 100 trees (10 folds x 10 bootstrapped resamples). Root nodes are the 1st node in a decision tree, and they are determined by which variable best optimizes a loss function (e.g., minimizes mean square error [MSE] for continuous outcomes or Gini Index for categorical outcomes). Put roughly, the most common root nodes can be thought of as the most “important” predictors.

# extract roots
bag_roots <-  function(x){
  x %>% 
  select(.extracts) %>% 
  unnest(cols = c(.extracts)) %>% 
  mutate(models = map(.extracts,
                  ~.x$model_df)) %>% 
  select(-.extracts) %>% 
  unnest(cols = c(models)) %>% 
  mutate(root = map_chr(model,
                     ~as.character(.x$fit$frame[1, 1]))) %>%
  select(root)  
}

# plot
bag_roots(fit_bag) %>% 
  ggplot(mapping = aes(x = fct_rev(fct_infreq(root)))) + 
  geom_bar() + 
  coord_flip() + 
  labs(x = "root", y = "count")

Random forest

Random forest is similar to bagged tree methodology but goes one step further. In addition to taking random subsets of data, the model also draws a random selection of features. Instead of utilizing all features, the random subset of features allows more predictors to be eligible root nodes. This is particularly useful for handling high dimensionality data (e.g., have more variables than participants/cases).

Specify the model

set.seed(100)
mod_rf <-rand_forest() %>%
  set_engine("ranger",
             num.threads = parallel::detectCores(), 
             importance = "permutation", 
             verbose = TRUE) %>% 
  set_mode("regression") %>% 
  set_args(trees = 1000)

Create workflow

wflow_rf <- workflow() %>% 
  add_model(mod_rf) %>% 
  add_recipe(rec)

Fit the model

set.seed(100)
plan(multisession)

fit_rf <- fit_resamples(
  wflow_rf,
  cv,
  metrics = metric_set(rmse, rsq),
  control = control_resamples(verbose = TRUE,
                              save_pred = TRUE,
                              extract = function(x) x)
)

Visualize

The plot below shows the root nodes from a random forest with 1000 trees (specified using set_args(trees = 1000) in the parsnip model object).

# extract roots
rf_tree_roots <- function(x){
  map_chr(1:1000, 
           ~ranger::treeInfo(x, tree = .)[1, "splitvarName"])
}

rf_roots <- function(x){
  x %>% 
  select(.extracts) %>% 
  unnest(cols = c(.extracts)) %>% 
  mutate(fit = map(.extracts,
                   ~.x$fit$fit$fit),
         oob_rmse = map_dbl(fit,
                         ~sqrt(.x$prediction.error)),
         roots = map(fit, 
                        ~rf_tree_roots(.))
         ) %>% 
  select(roots) %>% 
  unnest(cols = c(roots))
}

# plot
rf_roots(fit_rf) %>% 
  group_by(roots) %>% 
  count() %>% 
  arrange(desc(n)) %>% 
  filter(n > 75) %>% 
  ggplot(aes(fct_reorder(roots, n), n)) +
           geom_col() + 
           coord_flip() + 
  labs(x = "root", y = "count")

Boosted trees

Boosted trees, like bagged trees, are an ensemble model. Instead of applying successive models to resampled data and pooling estimates, boosted trees fit the next tree to the residuals (i.e., error term) of the prior tree. The goal is to minimize residual error through multiple trees, and is typically done with fairly “shallow” decision tree (i.e., 1-6 splits in each tree). Though each model is only slightly improving the error rate, the sequential use of many shallow trees makes computationally efficient (i.e. reduced run time) and highly accurate predictions.

Specify the model

mod_boost <- boost_tree() %>% 
  set_engine("xgboost", nthreads = parallel::detectCores()) %>% 
  set_mode("regression")

Create workflow

wflow_boost <- workflow() %>% 
  add_recipe(rec) %>% 
  add_model(mod_boost)

Fit the model

set.seed(100)
plan(multisession)

fit_boost <- fit_resamples(
  wflow_boost, 
  cv,
  metrics = metric_set(rmse, rsq),
  control = control_resamples(verbose = TRUE,
                              save_pred = TRUE)
)

Visualize

One of the few downfalls of {tidymodels} is its (current) inability to plot these tree-based models. For the past two models, it was simpler to extract root nodes and plot them, but their interpretation (as we’re fitting to residuals instead of data sets) are not straightforward. For that reason, we don’t have any pretty plots here. Instead, we’ll skip to evaluating the metrics of all models.

Evaluate metrics

After running these three models, it’s time to evaluate their performance. We can do this with tune::collect_metrics(). The table below shows the estimate of the out-of-sample performance for each of our 3 models.

collect_metrics(fit_bag) %>% 
  bind_rows(collect_metrics(fit_rf)) %>%
  bind_rows(collect_metrics(fit_boost)) %>% 
  filter(.metric == "rmse") %>% 
  mutate(model = c("bag", "rf", "boost")) %>% 
  select(model, everything()) %>% 
  knitr::kable()

model	.metric	.estimator	mean	n	std_err
bag	rmse	standard	98.42890	10	2.504904
rf	rmse	standard	95.20828	10	3.466279
boost	rmse	standard	95.35888	10	2.764773

Here, we are faced with a common problem in the machine learning world: choosing between models that perform similarly (see overlapping standard errors). Whether we would prefer random forests or bagged trees may depend on computational efficiency (i.e., time) or other factors. In practice, tuning several hyperparameters may have made one model clearly preferable over the others, but in our case - relying on all defaults - we would probably have similar performance with both models on a new data set and would prefer random forest or boosted tree models for their efficiency.

Out-of-sample performance

The final step is to apply each trained model to our test data using last_fit().

# bagged trees
final_fit_bag <- last_fit(
  wflow_bag,
  split = split
)

# random forest
final_fit_rf <- last_fit(
  wflow_rf,
  split = split
)

# boosted trees
final_fit_boost <- last_fit(
  wflow_boost,
  split = split
)

The table below shows the actual out-of-sample performance for each of our 3 models.

# show performance on test data
collect_metrics(final_fit_bag) %>% 
  bind_rows(collect_metrics(final_fit_rf)) %>%
  bind_rows(collect_metrics(final_fit_boost)) %>% 
  filter(.metric == "rmse") %>% 
  mutate(model = c("bag", "rf", "boost")) %>% 
  select(model, everything()) %>% 
  knitr::kable()

model	.metric	.estimator	.estimate
bag	rmse	standard	93.36504
rf	rmse	standard	91.18114
boost	rmse	standard	94.22609

After applying our 3 trained models to the unseen test data, it looks like random forest is the winner since it has the lowest RMSE. In this example, we only used 1% of the data to train these models, which could make it difficult to meaningfully compare their performance. However, the random forest model also results in the best out-of-sample prediction (RMSE = 83.47) when using all of the available data, which we did for the Kaggle competition.

Session Info

## ─ Session info ───────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.2 (2019-12-12)
##  os       macOS Catalina 10.15.6      
##  system   x86_64, darwin15.6.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       America/Los_Angeles         
##  date     2020-10-26                  
## 
## ─ Packages ───────────────────────────────────────────────────────────────────
##  package       * version    date       lib
##  assertthat      0.2.1      2019-03-21 [1]
##  backports       1.1.10     2020-09-15 [1]
##  baguette      * 0.0.1      2020-04-14 [1]
##  base64enc       0.1-3      2015-07-28 [1]
##  bayesplot       1.7.1      2019-12-01 [1]
##  blogdown        0.17       2019-11-13 [1]
##  bookdown        0.21       2020-10-13 [1]
##  boot            1.3-24     2019-12-20 [1]
##  broom         * 0.5.5      2020-02-29 [1]
##  C50             0.1.3      2020-01-13 [1]
##  callr           3.5.1      2020-10-13 [1]
##  cellranger      1.1.0      2016-07-27 [1]
##  class           7.3-15     2019-01-01 [2]
##  cli             2.1.0      2020-10-12 [1]
##  codetools       0.2-16     2018-12-24 [2]
##  colorspace      1.4-1      2019-03-18 [1]
##  colourpicker    1.0        2017-09-27 [1]
##  corrplot        0.84       2017-10-16 [1]
##  crayon          1.3.4      2017-09-16 [1]
##  crosstalk       1.1.0.1    2020-03-13 [1]
##  Cubist          0.2.3      2020-01-10 [1]
##  data.table      1.12.8     2019-12-09 [1]
##  DBI             1.1.0      2019-12-15 [1]
##  dbplyr          1.4.2      2019-06-17 [1]
##  dials         * 0.0.8.9000 2020-07-23 [1]
##  DiceDesign      1.8-1      2019-07-31 [1]
##  digest          0.6.27     2020-10-24 [1]
##  dplyr         * 1.0.2      2020-08-18 [1]
##  DT              0.16       2020-10-13 [1]
##  dygraphs        1.1.1.6    2018-07-11 [1]
##  earth           5.1.2      2019-11-09 [1]
##  ellipsis        0.3.1      2020-05-15 [1]
##  evaluate        0.14       2019-05-28 [1]
##  fansi           0.4.1      2020-01-08 [1]
##  farver          2.0.3      2020-01-16 [1]
##  fastmap         1.0.1      2019-10-08 [1]
##  forcats       * 0.4.0      2019-02-17 [1]
##  foreach         1.5.0      2020-03-30 [1]
##  Formula         1.2-3      2018-05-03 [1]
##  fs              1.3.1      2019-05-06 [1]
##  furrr           0.1.0      2018-05-16 [1]
##  future        * 1.18.0     2020-07-09 [1]
##  generics        0.0.2      2018-11-29 [1]
##  ggplot2       * 3.3.2      2020-06-19 [1]
##  ggridges        0.5.2      2020-01-12 [1]
##  globals         0.12.5     2019-12-07 [1]
##  glue            1.4.2      2020-08-27 [1]
##  gower           0.2.2      2020-06-23 [1]
##  GPfit           1.0-8      2019-02-08 [1]
##  gridExtra       2.3        2017-09-09 [1]
##  gtable          0.3.0      2019-03-25 [1]
##  gtools          3.8.1      2018-06-26 [1]
##  hardhat         0.1.4      2020-07-02 [1]
##  haven           2.2.0      2019-11-08 [1]
##  highr           0.8        2019-03-20 [1]
##  hms             0.5.3      2020-01-08 [1]
##  htmltools       0.5.0      2020-06-16 [1]
##  htmlwidgets     1.5.2      2020-10-03 [1]
##  httpuv          1.5.4      2020-06-06 [1]
##  httr            1.4.2      2020-07-20 [1]
##  igraph          1.2.4.2    2019-11-27 [1]
##  infer         * 0.5.1      2019-11-19 [1]
##  inline          0.3.15     2018-05-18 [1]
##  inum            1.0-1      2019-04-25 [1]
##  ipred           0.9-9      2019-04-28 [1]
##  iterators       1.0.12     2019-07-26 [1]
##  janeaustenr     0.1.5      2017-06-10 [1]
##  jsonlite        1.7.1      2020-09-07 [1]
##  knitr           1.30       2020-09-22 [1]
##  labeling        0.4.2      2020-10-20 [1]
##  later           1.1.0.1    2020-06-05 [1]
##  lattice         0.20-38    2018-11-04 [2]
##  lava            1.6.7      2020-03-05 [1]
##  lhs             1.0.2      2020-04-13 [1]
##  libcoin         1.0-5      2019-08-27 [1]
##  lifecycle       0.2.0      2020-03-06 [1]
##  listenv         0.8.0      2019-12-05 [1]
##  lme4            1.1-21     2019-03-05 [1]
##  loo             2.2.0      2019-12-19 [1]
##  lubridate       1.7.9      2020-06-08 [1]
##  magrittr        1.5        2014-11-22 [1]
##  markdown        1.1        2019-08-07 [1]
##  MASS            7.3-51.4   2019-03-31 [2]
##  Matrix          1.2-18     2019-11-27 [2]
##  matrixStats     0.56.0     2020-03-13 [1]
##  mime            0.9        2020-02-04 [1]
##  miniUI          0.1.1.1    2018-05-18 [1]
##  minqa           1.2.4      2014-10-09 [1]
##  modelr          0.1.6      2020-02-22 [1]
##  munsell         0.5.0      2018-06-12 [1]
##  mvtnorm         1.1-0      2020-02-24 [1]
##  nlme            3.1-142    2019-11-07 [2]
##  nloptr          1.2.2.1    2020-03-11 [1]
##  nnet            7.3-12     2016-02-02 [2]
##  parsnip       * 0.1.2.9000 2020-07-17 [1]
##  partykit        1.2-7      2020-03-06 [1]
##  pillar          1.4.6      2020-07-10 [1]
##  pkgbuild        1.1.0      2020-07-13 [1]
##  pkgconfig       2.0.3      2019-09-22 [1]
##  plotmo          3.5.7      2020-04-15 [1]
##  plotrix         3.7-8      2020-04-16 [1]
##  plyr            1.8.6      2020-03-03 [1]
##  prettyunits     1.1.1      2020-01-24 [1]
##  pROC            1.16.2     2020-03-19 [1]
##  processx        3.4.4      2020-09-03 [1]
##  prodlim         2019.11.13 2019-11-17 [1]
##  promises        1.1.1      2020-06-09 [1]
##  ps              1.4.0      2020-10-07 [1]
##  purrr         * 0.3.4      2020-04-17 [1]
##  R6              2.4.1      2019-11-12 [1]
##  ranger          0.12.1     2020-01-10 [1]
##  Rcpp            1.0.5      2020-07-06 [1]
##  readr         * 1.3.1      2018-12-21 [1]
##  readxl          1.3.1      2019-03-13 [1]
##  recipes       * 0.1.13     2020-06-23 [1]
##  repr            1.1.0      2020-01-28 [1]
##  reprex          0.3.0      2019-05-16 [1]
##  reshape2        1.4.3      2017-12-11 [1]
##  rlang           0.4.8      2020-10-08 [1]
##  rmarkdown       2.5        2020-10-21 [1]
##  rpart           4.1-15     2019-04-12 [1]
##  rsample       * 0.0.7      2020-06-04 [1]
##  rsconnect       0.8.16     2019-12-13 [1]
##  rstan           2.19.3     2020-02-11 [1]
##  rstanarm        2.19.2     2019-10-03 [1]
##  rstantools      2.0.0      2019-09-15 [1]
##  rstudioapi      0.11       2020-02-07 [1]
##  rvest           0.3.5      2019-11-08 [1]
##  scales        * 1.1.1      2020-05-11 [1]
##  sessioninfo     1.1.1      2018-11-05 [1]
##  shiny           1.5.0      2020-06-23 [1]
##  shinyjs         1.1        2020-01-13 [1]
##  shinystan       2.5.0      2018-05-01 [1]
##  shinythemes     1.1.2      2018-11-06 [1]
##  skimr         * 2.1.1      2020-04-16 [1]
##  SnowballC       0.6.0      2019-01-15 [1]
##  StanHeaders     2.21.0-1   2020-01-19 [1]
##  stringi         1.5.3      2020-09-09 [1]
##  stringr       * 1.4.0      2019-02-10 [1]
##  survival        3.1-8      2019-12-03 [2]
##  TeachingDemos   2.12       2020-04-07 [1]
##  threejs         0.3.3      2020-01-21 [1]
##  tibble        * 3.0.4      2020-10-12 [1]
##  tidymodels    * 0.1.0      2020-02-16 [1]
##  tidyposterior   0.0.2      2018-11-15 [1]
##  tidypredict     0.4.5      2020-02-10 [1]
##  tidyr         * 1.1.0      2020-05-20 [1]
##  tidyselect      1.1.0      2020-05-11 [1]
##  tidytext        0.2.3      2020-03-04 [1]
##  tidyverse     * 1.3.0      2019-11-21 [1]
##  timeDate        3043.102   2018-02-21 [1]
##  tokenizers      0.2.1      2018-03-29 [1]
##  tune          * 0.1.1.9000 2020-07-23 [1]
##  vctrs           0.3.4      2020-08-29 [1]
##  withr           2.3.0      2020-09-22 [1]
##  workflows     * 0.1.2.9000 2020-07-23 [1]
##  xfun            0.18       2020-09-29 [1]
##  xgboost       * 0.90.0.2   2019-08-01 [1]
##  xml2            1.2.2      2019-08-09 [1]
##  xtable          1.8-4      2019-04-21 [1]
##  xts             0.12-0     2020-01-19 [1]
##  yaml            2.2.1      2020-02-01 [1]
##  yardstick     * 0.0.7      2020-07-13 [1]
##  zoo             1.8-7      2020-01-10 [1]
##  source                               
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  Github (tidymodels/dials@fbea0c2)    
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  Github (tidymodels/parsnip@de6e9cd)  
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  Github (tidymodels/tune@8c280fd)     
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.2)                       
##  Github (tidymodels/workflows@d4cbe41)
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.0)                       
##  CRAN (R 3.6.2)                       
##  CRAN (R 3.6.0)                       
## 
## [1] /Users/bcullen/Library/R/3.6/library
## [2] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

Data Science Training in Psychology

Sun, 08 Mar 2020 00:00:00 +0000

Psychology is generally considered a “soft” science, on the opposite end of the spectrum from the “hard” sciences of physics, chemistry, and biology. This often creates a misconception that psychology is somehow not as quantitative, computational, or rigorous, methodologically. To the contrary, psychology is in fact very “hard” (some might even say, “the hardest”). Why? Because explaining human cognition and behavior is really difficult. People do not think and behave in predictable and consistent ways like atoms and molecules.

Indeed, the combination of random variation (“noise”) in how we humans respond to our environment on an given day/hour/minute, combined with the inherent challenge of proper sampling, reliable measurement and accurate model building makes data analysis for a psychologist no easy feat. Explaining variance in psychological data has required centuries of refining statistical methods and computational tools. Fortunately, today’s scientists have access to open-source software like R and Python that have vastly enhanced our ability to work with data in a way that is more transparent and reproducible. As a result, the teaching of these tools is becoming increasingly more common in psychology graduate programs (and many other fields), particularly in statistics classes.

Graduate statistics

The Department of Psychology at the University of Oregon requires first-year graduate students to take a 3-course series of classes on statistics and data analysis. (You can access the materials for these classes here.) As I’m sure used to be the case in psychology departments across many universities, the labs for these classes were originally taught in SPSS. However, thanks to the efforts of some hard-working graduate teaching assistants who served as lab instructors at the time, the labs eventually transitioned to being taught entirely in R.

When I started TA-ing for graduate stats, we (@roseberrymaier, @flourneuro) translated the labs from SPSS to R. We created a bunch of .Rmd files for students to work from. It's so cool to see that those students are the TAs now, and they're creating these amazing materials! https://t.co/myMGefLxQX
— Dr. Jessica Kosie (@JessKosie) March 6, 2020

While learning R in first-year grad stats has immense benefits, it is also really challenging. This was certainly my experience, anyway. In addition to grappling with the dense conceptual material, learning a programming language and using it to do statistics felt really daunting since I had very little programming background before graduate school. And the reality is, given that many (myself included) tend to experience a large learning curve with programming, combined with the already hefty gradstats curriculum, there just isn’t space in these classes to teach students much beyond the basics of R.

Training needs

But the way things seem to be moving, it’s becoming more common for many students to need more general data science skills in grad school. Current psychology grad students at UO (n = 28) were recently surveyed about the degree to which they perceive they will need different data science-related skills for their desired careers and how much training in those skills is available at UO to their knowledge. Specifically, they were asked,

On a scale of 0 (“Not at all”) to 4 (“Essential”), how much do you NEED this skill for your desired career?
On a scale of 0 (“Not at all”) to 4 (“Complete”), how much training is PROVIDED in this skill at your institution?

I created the plot below (using ggplot2) to visualize how much students report they need these skills compared to how much training they feel they already have.

Interestingly, the top two skills that students reported needing the most on average for their careers are data visualization and data wrangling. Fortunately, this was the first year (to my knowledge) that PSY611, the first course in the gradstats sequence, included entire labs devoted to data visualization and data wrangling. More good news is that other faculty in the psych department are starting to offer courses that focus more on these skills that students seem to need the most. For example, Rob Chavez has offered a fantastic seminar called Data Science Methods in Psychology that covered a wide range of data science topics from programming in R to text processing to package creation that ultimately turned into a great collection of student-created tutorials (all of which are available here).

A collection of fantastic R tutorials from my ‘Data Science Methods in Psychology’ seminar course, created by our amazing @UOPsych grad students. https://t.co/DhBcn940uz
— Rob Chavez (@robchavez) June 12, 2018

Among other crucial research skills like study design, data collection, and research ethics, the other most coveted technical skill seems to be programming. In fact, general programming skills seems to be where there is currently the biggest gap in training in our department. The graph below shows skills ranked in descending order by how large the gap is between how much training in those skills students need and how much they currently have.

On the whole, it seems that that there are bigger training gaps for more general data science skills (e.g., programming in R and Python, data wrangling, data visualization). In contrast, the smallest gaps exist for more specialized skills like advanced statistics (e.g., computational modeling, network analysis, structural equation modeling) – likely because our department has an entire class devoted to each of these topics.

Learning Resources

So, for the most part, it seems that grad students in our department want more training in general data science skills above and beyond the training they are currently receiving through coursework. Often what this means is that grad students end up teaching themselves (and each other) these skills. Some of my peers have made really incredible tutorials on new tools and methodologies they’ve recently learned. Sometimes students also attend specialized methods workshops and take really detailed notes that could greatly benefit others who want to learn the same kind of methods. My goal is to make it easier for people (both within our department and beyond) to find these resources.

To that end, I am currently working with Sara Weston to create and curate a searchable central repository of data science resources at UO that will house such student-created tutorials, workshop notes, and other data science products (e.g. R packages, shiny apps, etc.) that other students can benefit from. We are also planning to add resources from our department’s data science and statistics classes (e.g. the gradstats sequence, multilevel modeling, network analysis, data science seminar, etc.). The hope is that making these resources easier to find and share by putting them all in one place will start to reduce some of the current gaps in data science training. The additional benefit is that students will be able to share their data science “side projects” more broadly, and it will be easier for people to know who they might be able to ask for help with learning certain topics when they need it.

This is a work in progress, so stay tuned for more details!

A final thought

Reflecting on all this has brought home a thought that I think is worth sharing. I have come to realize that, as grad students, one of the best resources we have access to during our time in grad school is each other. Personally, I have learned so much over the years from my fellow lab members. Without them, I would not have conquered my fear of Github or felt inspired to learn more about R and other programming languages. Perhaps the most impactful benefit I’ve received from them, though, is their encouragement – and that, in particular, is something I hope to pay forward as much as possible.

We all enter grad school with different backgrounds and skill sets, and in so many situations the combination of our different strengths can lead to mutual benefit. So why not help each other out?

One Size Rarely Fits All

Sun, 08 Mar 2020 00:00:00 +0000

Think about the last time you had a splitting headache and needed some quick relief. Which bottle did you reach for? Tylenol or Advil? Why? Maybe you think of yourself as a “Tylenol person” or an “Advil person” because one of these has worked for you before. Or maybe one always upsets your stomach. The point is that not everyone experiences the same pain relief from one pill. Some people prefer Tylenol and others Advil, but nobody quite knows why.

The same is true of psychological treatments—they don’t work the same way for all people. People differ in how they respond to treatments for depression, anxiety, addiction, and so forth. The lingo in the field is that there are individual differences in treatment effectiveness. Why? Because, to state the obvious, no two people are exactly alike. And that fact poses a problem for the study of these treatments. Researchers test psychological treatments the same way they test drugs like Tylenol and Advil—using randomized controlled trials (RCTs). People are assigned at random to either one group, which receives a treatment, or another group, which does not. RCTs are an excellent way to assess the differences between the groups on average. The group difference indicates how well the treatment works. The problem is that researchers tend to treat people who differ from their group’s average as mere statistical fluctuations—“noise” in the signal.

The drawback of group-to-group comparisons is that they hide person-to-person differences that may be important. For example, even when one group does a lot better than another on average, there may still be many people in the first group who do not improve at all. Or maybe some people in the first group get worse even though their group got better overall. Instead of only thinking at the “group” level, we need to pay more attention to person-to-person differences. This will help us learn why treatments work only for some individuals and not others.

How do we learn for whom psychological treatments will work? There are three main things we need to know:

1) How does the treatment work?

At this point, there is strong evidence that Tylenol is an effective painkiller. But, even after decades of research, we still don’t fully understand how Tylenol relieves pain. Similarly, though we know that many psychological treatments are effective (at least for some people), we do not know how these treatments work. For example, researchers don’t know which parts of the brain are changed by most treatments. If we knew more specifics about how a treatment worked, we could improve it. We could focus our efforts on the parts of the treatment that make people better and remove the parts that don’t seem as important. And here’s the bonus: If we knew how treatments worked, that would help us learn for whom they work. Imagine we knew that a treatment reduced depression by decreasing self-criticism. In other words, a reduction in self-criticism is the one and only active ingredient in this treatment. If so, then people who criticize themselves a lot would be good candidates to benefit most from this treatment.

2) What are the unique causes of the problem for a given person?

To predict whether someone would benefit from a given treatment, we need to know more about the special factors underlying that person’s problem. In terms of pain, Tylenol is usually the better option for headache pain, and Advil is a good bet for inflammation. Similarly, a psychological treatment will work for a given person only if it addresses whatever caused the problem in the first place. Consider two people who both experience severe anxiety, but for different reasons. If genetic factors cause one person’s anxiety while past trauma is the main source of the other person’s anxiety, then these two individuals will likely need different types of treatment.

3) How does the treatment mesh with a person’s unique characteristics?

Like all drugs, Tylenol and Advil have side effects. And it turns out that they have different side effects depending on whether somebody is at risk for heart disease. So, people who have experienced a heart attack, for example, must carefully choose which pill to take when they have a headache. Similarly, whether or not a psychological treatment will work for somebody might depend on that person’s age, gender, or life history, to name just a few examples. The success of many treatments also depends on how severe a person’s problem is to begin with. Consider again the example of a treatment for depression that works on average for a group of people. Perhaps this treatment works best for people who are severely depressed because it focuses on a symptom that is more common in those people. A different treatment, which also works on average, might consistently decrease depression in mildly depressed people but not work at all for severely depressed people.

The bottom line

With answers to these three questions, researchers can begin to tailor psychological treatments to the unique needs of different people. We know that a given treatment will not work, at least not equally well, for every person. But being able to predict at the outset who will benefit and who won’t could save a lot of time and money. Rather than treating individual differences as “noise,” we can use them to glean potentially important information. If we can better understand why some people benefit from treatment while others don’t, we can make treatments more effective for more people. Eventually, this will allow more people to live healthier lives.

Note. I originally wrote this piece for Psychology Today. See the original post here.

Posts | Waseem Ashfaq

Flattening the leaRning curve: Teaching R online during COVID-19

Background

The challenge

Who attended?

Logistics

Zoom + Slack = 👌

RStudio Cloud

Blogdown site

Tools I learned

xaringanExtra

flair

flipbookr

countdown

Resources I recommend

Sharing on short notice

Teaching online on short notice

Teaching R online with RStudio Cloud

Teaching the tidyverse in 2020

Data science concept maps

rstats artwork

The importance of encouragement & positivity

Anticipate the challenge

Normalize errors & mistakes

Make your materials friendly and inviting!

Feedback

A final thought

Acknowledgements

Reflections on RStudio Instructor Training

The what

Training Course

Technical Exam

Teaching Exam

Demonstration Lesson: Example

Slides

Concept Map

Learner Persona

Formative Assessment

The why

Surge in online teaching

Teaching resources galore

Tried and tested

A focus on inclusivity

Community of practice

Get in touch

Tidymodels: Decision Tree Learning in R

Setup

Import the data

Explore the data

Split data and resample

Pre-processing

Create a model

The type of model

The engine

The mode

The arguments

Create a workflow

Model Examples

Bagged trees

Specify model

Create workflow

Fit the model

Visualize

Random forest

Specify the model

Create workflow

Fit the model

Visualize

Boosted trees

Specify the model

Create workflow

Fit the model

Visualize

Evaluate metrics

Out-of-sample performance

Data Science Training in Psychology

Graduate statistics

Training needs

Learning Resources

A final thought

`xaringanExtra`

`flair`

`flipbookr`

`countdown`