I'm Pat Schloss! I produce videos about how to use data science tools to answer questions about the world around me. I believe that anyone can answer their own questions. Do you?! I'd love to learn more about the world around you. Share the questions you would like to answer and we can take them on together!
Hello, which package or options did you use to check integrity of your code after passing it to styler ? I'm referring to the one you used via the Build button
It doesn't, but there is a separate GHA for running styler. We'll readdress this in the next episode so you can detect problems before pushing. I think the benefit of this particular GHA is for pull requests from others so that their code can get run through a lintr before trying to pull it into your code
Hello Patrick, I watched many of your videos and made my first R package to generate, filter, and sort relative abundance tables and make plots. We do a lot of amplicon sequencing from environmental samples at uWaterloo. The github repo is zjardyn/bubbler and the package is mostly done. I am still scared to submit it to CRAN lol.
Thank you for showing the benchmark about their performances (I still recommend you the `bench` package, though). How about `tidypolars` (in R, not in Python)?
@@Riffomonas thanks for sharing, I also tried myself to embeded my analytic code into a package.... but at some point I felt that generated data are too heavy for a R package. Also, I never considered (but I would like) using "test that" for an analytical workflow. In addition beyond "report generation" it would be nice to include github action within the workflow.
Something like Test Driven Development for data analysis is always rolling around in the back of my head :) It's hard because it requires using functions to test and most data analyses don't use home made functions, they use functions from other packages, which are hopefully already tested
It's about reproducibility and transparency. Modifying data in excel is very much not reproducible or transparent. It's next to impossible to document changes in excel files. Also, excel is $$$ whereas R is free. With R, we can document all of the changes in the script and rerun the script multiple times without having to worry about breaking things. Also, if I make a mistake, I can easily correct it by reprocessing the raw data with the corrected code.
despite not currently developing an R package or planning to, i really enjoy watching your series on making one especially since you're showing the backstage, raw and "ugly" side of coding (the typos in particular). if it ever comes to me creating a package i'll make sure to use this series as a reference. cheers!
I dont like my axis labels to repeat themselves. Is there a way to have a single centered x and or y labelling. Been trying to figure it out but cannot manage to get it rght
You can get it through the links in the blog post linked in the show notes. github.com/riffomonas/climate_viz/tree/a428f64b7db493145bf84ec6f38f8e89da258675/data
You can certainly grow the list, which really isn't a problem if you don't think it will be long. Alternatively, I've also seen people initialize a list that's larger than you think it will be and then prune it after you know how big it actually should be
Thank you for the video. I just have a question about the @examples section. When I preview the documentation file under the help tab in Rstudio, I see a link that says "Run examples." When I click this link, it takes me to a blank page that says "Example/<function name> not found." When I look at other packages, I can click on this link, and it shows the output given by the examples. Is there something I have to do to get this link to work? Or does this happen after the package is submitted?
Sir, I have always been thrilled by your R techniques and the way of your explanation. Your videos have caused me to switch to R [and of course RStudio] from Python [Jupyter] literally captivated me in enjoying analytics using R. I am working in Aerospace Industry. While the organization leverages a couple of premium visualization tools even to analyze exploratorily, I, after having got lightning stuck by your amazing videos, have been using ggplot extensively along with my basic statistical knowledge and of course teaching my colleagues. Thank you, @Riffomonas. Please keep posting more videos to enlighten some of thirsty analysts like me.
Wow! Thanks Pat. This was the most advanced episode in the series and it requires one to be well versed in the intricacies of base R. Thoroughly enjoyed it!
Great instructional video, thanks! During the pandemic I became proficient at this but haven't used Git in a couple of years. You provided just the guidance I needed.
One of the frustrating limitations of the conda approach for R is that only a fraction of the packages in CRAN are compiled and installable via conda. Other packages are not in CRAN and its not clear how to install those via conda. Any thoughts on this?
I have found that the major packages are available in one of the conda collections. It's not a horrible process to contribute a conda package if you need to. This SO link is somewhat helpful stackoverflow.com/questions/52061664/install-r-package-from-github-using-conda
Three years since this video was posted, and here I am too making a nice figure. And this is not the only video that has helped me. Thank you very much, Dr Schloss.
Hi Pat. thank you for this could you please check the blog post? I guess it is not uploaded yet thank you so much for the knowledge sharing and efforts you do which have helped me immensly
Hey, if your local community uses Julia, go for it. Same goes for Fortran, Haskell, whatever. There's no debate. The only rule is that people need to learn to program. It's best to learn what your local community uses. No interest in engaging in any type of language wars here 🤓
Thanks! I think I got the @returns after editing the video :) It really doesn't matter if you use @return or @returns, but @returns is the new way of doing things.
I wonder how the process today might look if you're doing something like this with ChatGPT: Providing the function to chatgpt Uploading roxyegn documentation (optional) Asking it to write documentation with the important key headers, including examples. That way you can make sure your wording is the same across functions, as well as argument names. I mean, must provide some context but I bet it can spare a lot of time
I'll make a deal with you and all of my other viewers... I promise that I will never intentionally use ChatGPT or its ilk to generate code, documentation, anything on my channel :)
Just to note, using as.data.table() or setDT() will be considerably faster than data.table(). data.table also comes with its own version of merge() so you don't have to use the funky syntax for a full merge.
@@Riffomonas I made synthetic datasets since I don't have your fasta data. In my test, dtA was the fastest, followed by using setDT and as.data.table. data.table() was similar to dplyr with inner_join. Here is my code: each_num <- 1e4 animal_legs <- map_dfr(data.frame(animal = c("cow", "fish", "chicken", "dog", "sheep"), n_legs = c(4, 0, 2, 4, 4)), rep, each = each_num) %>% mutate(n = 1:nrow(.)) animal_sounds <- map_dfr(data.frame(animal = c("cow", "chicken", "cat", "sheep", "dog"), sounds = c("mooo", "cluck", "meow", "baaa", "bark")), rep, each = each_num) %>% mutate(n = 1:nrow(.)) # make a copy for setDT because it changes things in place, # which would affect everything else relying on animal_legs and animal_sounds # in microbenchmark. animal_legs_2 <- copy(animal_legs) animal_sounds_2 <- copy(animal_sounds) #data.table for dtA test animal_legs_dt <- data.table::data.table(animal_legs, key = "n") animal_sounds_dt <- data.table::data.table(animal_sounds, key = "n") microbenchmark::microbenchmark( base = base::merge(animal_legs, animal_sounds, by = "n", all = FALSE), ij = dplyr::inner_join(animal_legs, animal_sounds, by = "n"), dt = { animal_legs_dt_test <- data.table::data.table(animal_legs, key = "n") animal_sounds_dt_test <- data.table::data.table(animal_sounds, key = "n") animal_legs_dt_test[animal_sounds_dt_test, nomatch = NULL, on = .(n)] #inner join }, dtA = animal_legs_dt[animal_sounds_dt, nomatch = NULL, on = .(n)], as_dt = { animal_legs_dt_test <- data.table::as.data.table(animal_legs, key = "n") animal_sounds_dt_test <- data.table::as.data.table(animal_sounds, key = "n") animal_legs_dt_test[animal_sounds_dt_test, nomatch = NULL, on = .(n)] }, set_dt = setDT(animal_legs_2)[setDT(animal_sounds_2), nomatch = NULL, on = .(n)] ) Here's the results table on my computer: Unit: milliseconds expr min lq mean median uq max neval cld base 29.754131 31.562985 33.248288 32.509195 34.936214 38.573338 100 a ij 3.847486 4.260528 5.063156 4.359981 4.581972 10.221760 100 b dt 3.840432 4.029632 6.483946 4.263007 4.815313 145.038057 100 b dtA 2.474058 2.599248 4.473535 2.822705 3.102000 137.916202 100 b as_dt 3.255442 3.403854 4.397897 3.637299 4.121683 11.064186 100 b set_dt 2.881791 2.995842 3.809361 3.250692 3.767623 7.726401 100 b Base R was actually the fastest when I did this test with the original animal_* datasets, but it clearly doesn't scale very well.
@@Riffomonas Apologies if this is showing up for a second time, but I replied earlier and now it seems to be gone. I made some synthetic data because I don't have your fasta data. I consistently find that dtA is fastest, followed by setDT() and as.data.table(). dplyr and data.table() are comparable. Here is my code: each_num <- 1e4 animal_legs <- map_dfr(data.frame(animal = c("cow", "fish", "chicken", "dog", "sheep"), n_legs = c(4, 0, 2, 4, 4)), rep, each = each_num) %>% mutate(n = 1:nrow(.)) animal_sounds <- map_dfr(data.frame(animal = c("cow", "chicken", "cat", "sheep", "dog"), sounds = c("mooo", "cluck", "meow", "baaa", "bark")), rep, each = each_num) %>% mutate(n = 1:nrow(.)) # make a copy for setDT because it changes things in place, # which would affect everything else relying on animal_legs and animal_sounds # in microbenchmark. animal_legs_2 <- copy(animal_legs) animal_sounds_2 <- copy(animal_sounds) #data.table for dtA test animal_legs_dt <- data.table::data.table(animal_legs, key = "n") animal_sounds_dt <- data.table::data.table(animal_sounds, key = "n") microbenchmark::microbenchmark( base = base::merge(animal_legs, animal_sounds, by = "n", all = FALSE), ij = dplyr::inner_join(animal_legs, animal_sounds, by = "n"), dt = { animal_legs_dt_test <- data.table::data.table(animal_legs, key = "n") animal_sounds_dt_test <- data.table::data.table(animal_sounds, key = "n") animal_legs_dt_test[animal_sounds_dt_test, nomatch = NULL, on = .(n)] #inner join }, dtA = animal_legs_dt[animal_sounds_dt, nomatch = NULL, on = .(n)], as_dt = { animal_legs_dt_test <- data.table::as.data.table(animal_legs, key = "n") animal_sounds_dt_test <- data.table::as.data.table(animal_sounds, key = "n") animal_legs_dt_test[animal_sounds_dt_test, nomatch = NULL, on = .(n)] }, set_dt = setDT(animal_legs_2)[setDT(animal_sounds_2), nomatch = NULL, on = .(n)] )
@@Riffomonas I've tried replying a few times, but youtube seems to be autoremoving the comment. Maybe something to do with the code snippet.... Anyway, I made large synthetic datasets because I don't have the fasta data, and ran everything again. dtA is consistently fastest, followed by setDT and as.data.table(). Here is my code below. each_num <- 1e4 animal_legs <- map_dfr(data.frame(animal = c("cow", "fish", "chicken", "dog", "sheep"), n_legs = c(4, 0, 2, 4, 4)), rep, each = each_num) %>% mutate(n = 1:nrow(.)) animal_sounds <- map_dfr(data.frame(animal = c("cow", "chicken", "cat", "sheep", "dog"), sounds = c("mooo", "cluck", "meow", "baaa", "bark")), rep, each = each_num) %>% mutate(n = 1:nrow(.)) # make a copy for setDT because it changes things in place, # which would affect everything else relying on animal_legs and animal_sounds # in microbenchmark. animal_legs_2 <- copy(animal_legs) animal_sounds_2 <- copy(animal_sounds) # data.table for dtA test animal_legs_dt <- data.table::data.table(animal_legs, key = "n") animal_sounds_dt <- data.table::data.table(animal_sounds, key = "n") microbenchmark::microbenchmark( base = base::merge(animal_legs, animal_sounds, by = "n", all = FALSE), ij = dplyr::inner_join(animal_legs, animal_sounds, by = "n"), dt = { animal_legs_dt_test <- data.table::data.table(animal_legs, key = "n") animal_sounds_dt_test <- data.table::data.table(animal_sounds, key = "n") animal_legs_dt_test[animal_sounds_dt_test, nomatch = NULL, on = .(n)] # inner join }, dtA = animal_legs_dt[animal_sounds_dt, nomatch = NULL, on = .(n)], as_dt = { animal_legs_dt_test <- data.table::as.data.table(animal_legs, key = "n") animal_sounds_dt_test <- data.table::as.data.table(animal_sounds, key = "n") animal_legs_dt_test[animal_sounds_dt_test, nomatch = NULL, on = .(n)] }, set_dt = setDT(animal_legs_2)[setDT(animal_sounds_2), nomatch = NULL, on = .(n)] )
@@Riffomonas I've tried replying numerous times, but my comment gets removed each time. I think it doesn't like the code snippet I'm trying to share... Anyway, I made synthetic datasets ~50,000 rows long, where each row is a unique group so that it is comparable to your fasta data. dtA is consistently fastest, followed by setDT and as.data.table. One thing I had to control for was using a copy of the dataframe for setDT (e.g., df_copy <- copy(df)) because setDT works in place. If you use the same df reference for all items in microbenchmark, you run the risk of setDT changing your dataframe in place to a data.table. Then any subsequent runs with data.table() or as.data.table() take the same amount of time because df is already a data.table. Maybe youtube will let me share the code as a github link... : github.com/mrguyperson/joins_example/blob/main/R/joins.R
A few thoughts come to mind. 1) dplyr always outputs tibbles. If you're going to use dplyr, it might be worth using tibbles throughout your package. The loss in performance is worth the consistent formatting, and tibbles are just better. 2) dplyr allows for multiple backends (dtplyr, dbplyr, duckplyr, arrow, etc). Would those affect your code? If I call duckplyr::methods_overwrite(), and a package has a custom function that calls dplyr::inner_join() under the hood, would it now call duckplyr::inner_join() under the hood instead? 3) Similarly, If I pipe a dataframe into dtplyr::lazy_dt() and then into a custom join function that calls dplyr::inner_join() under the hood, would it work and use the data.table method? Or would dtplyr just not know how to translate the code? I know that you're not planning to write a custom join function, but your video still sparked these curiosities in me. Lately I've been looking at these dplyr backends as a way to scale up our work for big data projects without making my team have to learn new syntax, so they've been on my mind a lot. Great video as always!
Great - thanks for the feedback. For now, the input to the phylotypr functions will be data.frames, but they should work fine if people provide tibbles or data.tables. The output will be base R structures like lists and character strings.
That was super helpful, thank you! I'll admit that joining was something that I hadn't really got the hang of, and even though I have gone through the tutorials, I didn't really appreciate what was going on. Only got rolling to figure out and then I can say I've mastered data.table! I would add though that the animal_legs_dt[animal_sounds_dt[uniq_animals]] for a full join is... pretty ugly! Instead, data.table provides its own version of merge that looks exactly like base R.
Excellent video as always! You might want to check out the tidytable package by Mark Fairbanks. It provides a fast data.table backend for many dplyr, purrr, and tidyr functions, with tidyverse syntax. I find it works great for speeding up R package development and code, while keeping dependencies down and keeping the code readable/maintainable for those not as familiar with data.table.
I have been trying this and some URL exit without the URL in single quotes and some need single quote URL to exit. Gone to encapsulating all URL in single quotes.
yeah bash can sometimes do different things with single vs double quotes. a backslash can be useful for escaping quotes if you need to have quotes in quotes
Hi Pat, thanks for the nice vedio! when use nmds <- metaMDS(shared, autotransform = FALSE), then score(nmds), the output has both $sites (which is the Group here) and $species (OTUs). I cannot directly pipe it to ggplot. I wonder how you deal with it? Thanks!
Hmmm, I'm not sure - why are you giving metaMDS shared instead of a distance matrix? Could that be the difference between what you and I are doing? github.com/riffomonas/distances/blob/main/code/nmds.R
Thank you for the guidance. Do you have a video on performing Principal Component Analysis on microbial data? It will be helpful to see you do this. I am learning a lot from you, Prof.
Thanks for watching check out these two videos: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-G5Qckqq5Erw.html and ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-h7OrVmT7Ja8.html
