Panel Data (Fixed Effects, Random Effects) - R for Economists Moderate 9 

Two years later it is still helping with master's thesis.

i know Im asking randomly but does someone know a way to log back into an Instagram account..? I was dumb lost my login password. I would appreciate any help you can offer me!

@Michael Kyrie instablaster :)

@Bentlee Alessandro I really appreciate your reply. I found the site through google and Im trying it out atm. Seems to take a while so I will reply here later with my results.

@Bentlee Alessandro it did the trick and I actually got access to my account again. Im so happy:D Thank you so much you saved my account!

@Michael Kyrie You are welcome :D

I'm afraid I don't have materials on those except or partial F (see my post regression statistics video), but for vif I recommend the jtools package. Never done a VAR in R myself I'm afraid

In plm, lag(x, 2) will lag 2 periods for example

Thanks! I'd recommend checking out this section of my fixed effects chapter www.theeffectbook.net/ch-FixedEffects.html#random-effects

I don't think plm can do it directly but you can compute it by hand as in karthur.org/2016/fixed-effects-panel-models-in-r.html Or, estimate it instead in lfe (see the same link) or estimatr (see my estimatr video) which report the full R squarws

I'm afraid I don't have any videos on missing data. The standard thing to do is just to drop observations with missing data, although that introduces a fair number of problems. You may want to look into multiple imputation.

Thank you very much! There are some small differences in methods (often computational differences as opposed to statistical differences) when it comes to unbalanced panel data. However, conveniently, most main-line panel data software is designed to handle it already, plm being an example of a package that does so. So you can use the same commands with rare exception.

@@NickHuntingtonKlein Thx so much!

Yes, that should be true as long as you have exactly two time periods. In plm I'm getting numerically identical results for effect = "twoways" but not effect = "individual". i.e. you add in a time period dummy to the within model and it works. That lines up with what I get using regular lm() on demeaned data. I haven't thought about that equivalence in a while so I can't remember if a time dummy is a part of it, but that's what I'm getting.

I've never done it myself, but I believe the package to do it in is pgmm

@@NickHuntingtonKlein Alright thank you, I'll have a look!

If you want to add year effects you're better off by specifying it as a two-way model in the options

@@NickHuntingtonKlein can you please point me in the right direction. I didn't get you, what do you mean by two way model

@@rajat1770 A two way fixed effects model is a regression model with fixed effects for individual/group and also fixed effects for time. You can get this in plm using effect = "twoways" as long as you specify both a group and time variable for the index

Glad it helped! The standard test is the variance inflation factor or VIF. I don't know the R command off the top of my head but there's your Google term.

@@NickHuntingtonKlein Thanks for replying so quickly. The VIF commands only seem to work for lm not plm commands. Anyway, thanks for taking the time, it's much appreciated.

For all of these I'd recommend checking out my video on the fixest package, which makes two way fixed effects easy and has an easily customizable lag function

@@NickHuntingtonKlein Thank you, sir!

Thanks! I'm afraid I don't have any blp materials though.

Depends what kind of visualization you want! jtools has some good regression visualizations. I've also been largely using fixest for fixed effects these days, and it has its own set of regression visualization functions

I'd call that a repeated cross section. You can add some sorts of fixed effects to that (like time) but not others (like individual). If plm won't let you register it as a pdata.frame, check out my estimatr video

The vcovDC() function in plm should be able to do it. But also you could switch to using the fixest package instead which has easy syntax for doing this.

Insignificant results doesn't mean the model is wrong, it just means your results are insignificant! That said, FE does suck a lot of the statistical power out of a model, especially if you have a "big-N-small-T" model (i.e. many groups but few observations per group). If you're pretty certain that your predictors *should* be predicting a large share of the within-variance but you're not getting anything, consider if maybe you are actually controlling away more variation than you intend. Ask: after removing the variation from the fixed effects, should there be enough variation left to actually study? You might also try random effects if you think those assumptions might be likely to hold.

Including dummy variables in panel regression is totally fine (just be careful that they don't overlap too much with your fixed effects). The meaning of FE is slightly different in logit-FE or probit-FE than in OLS-FE, but the same general ideas work. The interpretation of the coefficients will be the same as in *logit or probit* models, not the same as in OLS-FE. I should point out that the "1% in log value" interpretation is for use with logged variables, not with logit/probit - those are different.

Yes, specifying it within plm() is the way to go - this should produce the same results as including year as a set of dummies, except perhaps for any standard error adjustments, and in that case the version with both included (and 'twoway' fixed effects specified) would be preferred. As for the NAs, if there are NAs in the data, plm will be dropping them anyway. How much damage this does to the results depends on how much data is missing and why. See the section on missing data in the Under the Rug chapter in my book nickchk.com/causalitybook.html

@@NickHuntingtonKlein Thank you so much Nick!

I'd recommend checking out my video on fixest, that package makes this fairly easy

I do have some videos on EDA in my Data communications series

@@NickHuntingtonKlein Great. Would you mind to share the link for those videos on EDA? By the way, I like to read Economics papers from top Economics Journals such as AEA, QJE..NBER, when I see they are written, analyzed, do they use Rmarkdown, Quarto, Stata Markdown..LaTex or Overleaf? What is the typingsetting system recommended for Economists with the mind of reproducibility and replication? Thanks.

@@kwizeralambert1316 apologies, looks like I added the EDA section to that class after making those videos. But you can see the updated course, including the EDA lecture, here github.com/nickch-k/datacommslides Most economists use either latex or Word. Markdown is slowly gaining popularity.

@@NickHuntingtonKlein Thank you so much, Very rich courses. I understand, LaTex is widely used. Currently, I see some some researchers are using Quarto and Rmarkdown to write their papers

@@kwizeralambert1316 yes I'm pretty much exclusively using quarto from here on out, outside of collaboration with people who don't use it

Try running some lmer output (from lme4) through coeftest from the sandwich package, or just send it to msummary (in the modelsummary package) and set the SE type with the vcov option.

@@NickHuntingtonKleinThanks! btw, What is the difference between this RE (Robust SE) from RE in your video ?

@@joseluissola8941 RE means random effects, not Robust se. So they're different things entirely

After using xtset to declare the panel structure, use xtreg to run fixed or random effects. Use L. In front of a variable to lag it.

@@NickHuntingtonKlein Could you please advise, how to generate 1lag dependent variable (as you have it in your video)? I tried with slide command, and setDF command, ...etc... and nothing really works... additional problem is that I have some missing values...

@@macro_finance in stata? Just use L. on the dependent variable. Or if you're back in R, plm::lag should do it

@@NickHuntingtonKlein No, sorry, it is in R. Actually, whenever I run the fixed effect estimator test in R, I get the error message: Error in class(x)

@@macro_finance did you use pdata.frame before running fixed effects, like in the video?

If you want both effects, then yes you'd want to include fixed effects for both

@@NickHuntingtonKlein thanks you for the video and the quick answer (nice and complete)

That won't work. Fixed effects for logit work very differently than for linear models. Check out the bife package for fixed effects logit. I haven't used it myself but I hear good things.

@@NickHuntingtonKlein Thank you, I will look into how to use that package! I have been having trouble finding information on how to determine weather I should be using a FE model or RE model for my logistic regression. Is the Hausman test the best tool to use when making this determination?

@@onigiriman Hausman is designed for linear models, so that wouldn't work, unless there's some special logit version. I'm afraid I don't know too much about logit random effects.

@@NickHuntingtonKlein I see, thank you so much for your replies. They are very helpful!

I'm afraid I don't know French, but it sounds like an internal problem with plm maybe? I'd recommend asking on StackExchange.

as.factor() will turn country name into a factor variable, which will be acceptable for use with plm (and other panel packages) as an id variable

For Stata I'd recommend looking into their panel commands. xtset lets it know you're working with panel data, and then you can do fixed effects with xtreg

@@NickHuntingtonKlein thank you professor for your interest . either not professional but I feel like crashing with my model but I really thankful to your reply.

It's correct that you can't add industry as a control when you already have firm fixed effects, since they'd be perfectly collinear. The good news is you don't need to! Including firm fixed effects automatically controls for anything that's fixed within firm, like industry. So you don't need to add the industry controls, that job is already done by the firm fixed effects. If your goal is to study the effect of industry itself, are you sure you really want to control for firm? What identification problem does that solve for you? If you do really want both you'll need to run some sort of hierarchical random effects model, which can get tricky. The relevant R package for that would be lmer.

@@lanaschludi5466 if it says they did industry and year fixed effects then they probably didn't do firm fixed effects, just industry and year. The "yes" in the table just means "yes, we included these fixed effects, but the estimates for them aren't important so we're not going to show them to you, we just included them as controls." so you can ignore firm FE and just do industry and year.

In plm it should just work automatically. It assumes that the levels of the time variable it finds in the original data are consecutive and ascending.

@@NickHuntingtonKlein Thanks a lot for the feedback. I have tried adjust my index variable, however, I have the error: "In pdata.frame(Data) : duplicate couples (id-time) in resulting pdata.frame to find out which, use, e.g., table(index(your_pdataframe), useNA = "ifany")."

@@godwinezekoye211 sounds like you have duplicates. Did you run the check it said? Plm won't work if you have multiple observations per id/time. If you want that (and are just planning to do time FEs, obviously lags won't work in this setup), switch from plm to fixest, see my fixest video.

You have to have multiple observations of each of the things you have fixed effects of. So you can't have individual-level fixed effects in cross-sectional data. You could have fixed effects for some higher-level grouping. So for example if you had a cross-sectional data set of people, you can't have fixed effects for people, but you can have fixed effects for, say, city, as long as multiple people in your data are in each city. To do this in R, just add +factor(city) to your regression model, where city is the variable you want to add fixed effects of.

same problem. how did u fix it?

Depends on how crime rate is defined. Old crime rates determine current police presence which determines current crime rate. So with the variables in the data, you can consider "lagged crime rate" to be an omitted variable. But yes your interpretation makes sense too.

Sure, if it's brief

thankyou! I have purchase panel data of consumers over a period of time (1.5 year). Within this period of time there is an intervention of +/ - 0.2 years implemented. this purchase data consist out of the purchase of products. These products can belong to a productgroup (in total 10 productsgroups) I am interested in the effect of the intervention on the sales of the product for each category (and compare them?). Furthermore, i do have some interaction variables with the intervention variable, so i can analyze what the effect of these variables is during such intervention As i have panel data for each invidual it is the most powerfull to determine this effect at an invidual level. However, i am not sure how to determine if i should use a fixed, a random or mixed model. Can someone give me suggestions/advice how to approach this?

@@rutgervanbasten2159 In your case, since you're interested in how the effect varies over different predictors, I might recommend a mixed model, and specifically an HLM where you model the effect of the intervention as a function of your predictors.

It's hard to say too much without knowing your data, but it sounds like you have multiple observations per combination of country/year, which plm doesn't like. I'd recommend checking out my video on the estimatr package and using that instead.

See this page on how to use the pht function, or how to run H-T in plm: rdrr.io/cran/plm/man/pht.html

@@NickHuntingtonKlein thank you. I was trying to understand how they did the classification of the variables with such a simplified command but I found it explained better in another site.

The 540 is only for the first-differencing model, since it by necessity has to drop one of the time periods. Is that the one you're running?

@@NickHuntingtonKlein no sir , I runned the ( #include a lag ) model and from this ideo, it became 540 , while in my rstudio its 629. Apparently, I found the problem sir. When I load dplyr package, my result for the # include a lag model is N = 629. Once I unloaded dplyr and run it again, it becomes 540 (probably from 630- 90 county) like your video. However, i don't know why dplyr could do this 😂

@@christina-4287 oh, yeah, one of the downsides of dplyr is that it shares function names with a lot of other functions, so sometimes loading it will overwrite a necessary function you need

@@NickHuntingtonKlein okayy so is it better to unload the dplyr in this case 👍. Moreover sir, I tried to implement it to my data. I already unload the dplyr, and i have an original N= 1187. But when i regressed it using the lag of Y, the N drop very drastically to N = 836. But my county only 68 firms. I thought it should be 1187 - 68 only but it's not... There is also no missing value.. so why do you think R exclude many obsv in the regressiion? Thank you 🙏

​@@christina-4287 You don't necessarily hav eto unload dplyr, but you want to be sure to use the plm version of lag() so instead of lag() you can just say plm::lag(). As for the data dropping rapidly, are there perhaps gaps in the data? If you have time periods 1, 3, 4, 5, then applying a lag will drop both 1 and 3.

Glad you like it! I'm afraid I don't know a good explainer for this. My best bet would be to just Google each term you're unfamiliar with

@@NickHuntingtonKlein ... Well ok :'( thank you very much for your quick answer. I gonna work!

A natural experiment is when you find a source of close-to-exogenous (randomized) variation in the real world without a researcher running the experiment themselves. For example, the US Vietnam draft lottery was administered by randomly ordering birthdates. So your likelihood of being drafted into the military was random, and that could be used as a sort of experiment to look at the effects of being in the military on later outcomes.

For any R function you can get the help file with help(). So in this case, help(lag). You'll want to select the one titled "lag a time series"

@@NickHuntingtonKlein ok, so if i want a secon lag in my plm, than i type plm(y= lag(y,k=2) +x...)

@@mulle171 correct, that will give you a two period lag. If you want both one and two period lags you'll need two separate lag functions though

amazing :D is there also a way to optimize the amount of lags ? i would have thought maybe up to the point where the variables are significant but the r^2 is as low as possible

@@mulle171 Typically you would try a bunch of different lag lengths and select the one that gives the best AIC or BIC statistic (Aikake/Bayesian Information Criterion). In general you pretty much never want to use statistical significance to make decisions about your model.

Usually the same software tools work; the necessary adjustments for unbalanced panels are often automatic and included. Check the documentation of your command of choice though

@@NickHuntingtonKlein I think the problem is there duplicate firm-year ids but I always eliminate the duplicates. I matched my outomce variable with a different firm identifier.. could it be that?

No, it is actually better the way he is doing it, gives a lecture feel to the video and makes it more engaging. Thanks for the great content Nick!

@@juanpabloaguirre6390 It's actually very distracting and annoying. Guess we all learn differently