Great Video.. This was really needed. There are Videos on AWS Glue and Glue Data catalogs and some of them show only the basic operations which can be done in glue. But this video clearly explains how we can implement Complex ETL transformations in Glue using a combination of Pyspark and glue Syntaxes. This video is closest to real world scenario where we need to implement complex data transformations.
Thank you very much for the knowledge, this was very useful. Can we drop the Glue DynamicFrame from the memory after we have converted it to the Spark DataFrame? In order to reduce the memory usage. Since the DynamicFrame is just taking up space. Thank you
Getting this error importing provided yaml - The following resource types are not supported for resource import: AWS::Glue::Database,AWS::Glue::Table,AWS::Glue::Table,AWS::Glue::Table,AWS::Glue:🏓
MMM i have an error with IAM role. Running the notebook cell gives: Exception encountered while creating session: An error occurred (AccessDeniedException) when calling the CreateSession operation: Account ----- is denied access.
Hi, My IAM role has both of these roles, but I get the following error when trying to run the second block in my notebook: An error occurred (AccessDeniedException) when calling the CreateSession operation: User: assumed-role/AWSGlueServiceRoleDefault/GlueJobRunnerSession is not authorized to perform: iam:PassRole on resource: AWSGlueServiceRoleDefault because no identity-based policy allows the iam:PassRole action. What did I do wrong?
amazing video! learned a lot, thanks man. By the way is there a setting I can use to activate code suggestions in glue notebook? Also is it correct that it is billed per notebook session duration and not by number of run?
Thanks for the tutorial, Johnny, it's really good. I paid attention that show() method of the Glue data frame doesn't use the parameter at all (always defaulting to the first 20 both in your video and in my cluster as well). if however, you convert it to Spark DataFrame, it works like a charm. Not a big deal in this case, but now I'm not sure how confident I am to run it in the prod...
Great tutorial and very clear. I get the forllowing error running it, Exception encountered while creating session: An error occurred (AccessDeniedException) when calling the CreateSession operation: Account xxxx is denied access. I confirmed in IAM the role was created properly as defined in the cloudformation template. I'm using N.Virginia region. Please help.
Great work Johnny, so helpful ! However I have a question please : "How to create a dynamic frame using an existing jdbc connector (in the data catalog) and a custom sql string query (not only a table, complicated query) ?"
Hey Johnny! Thanks for making such great quality tutorials, I've learned a ton! As a side note, I'm fascinated with your names for symbols, as I've never heard anyone refer to them as you do. I did a double-take every time you said "curlies". My names are: ( ) -> parentheses, or "parens" (you call them curly brackets) { } -> curly braces, or "curlies" (not sure what you call these) [ ] -> square brackets (also not sure) < > -> angle brackets (also not sure) Is this a regional thing? Similar to "." being a "period" to me, and a "full-stop" to you?
Sir, your video is awesome… I was struggling very badly to learn AWS, now I have become expert by watching your video and able to write my own scripts… Really Thanks a lot…
Thanks Johnny ! This is great I have a very specific scenario where I wish to develop pyspark code locally (on my laptop) -> package it (egg or zip) -> deploy on s3 -> trigger from AWS Glue A questions: As I am not using Glue dynamic frame and written my code in pure pyspark format, can I still use AWS Glue catalog as input & output or would I have to R/W directly from S3 ?
Thanks for sharing this. May I know how did you create IAM role for this and what are the policies you have attached to it? I don’t see it during the start of the video where you create notebook session
For the notebook itself? I created the role using the cloud formation template that we use to spin up all the resources once we logged into aws on the video. You can view the code file on GitHub where you’ll see the IAM/policies defined.
Hey kishlaya, all the videos on the channel which cover AWS glue are outside of the free tier. If you generally stay within the free tier, and this is reflected in our monthly account bill then open a support ticket with AWS immediately. Explain you where following an online glue tutorial and didn’t realise it would involve a charge of services. They maybe able to help you, especially if it’s a significant amount of money to you personally. AWS are very customer centric.
Johnny! You are a knowledgable excellent teacher! You really lay things out perfectly and make it fun to learn. I have challenges in my own ETL jobs that I hope you might address in a video someday. It would be nice if you show some advanced techniques for writing to JDBC and Postgres where the target database has data types such as uuid or enumerated data types. I would also like to know the best way to do an upsert (if the record is new do an insert otherwise do an update). Thanks again!
This was an amazing tutorial. I understood every bit of it because of the way it was explained with hands-on. Loved hand typing of all commands which seemed very real world scenario. Thank you so much Johnny!
Great content man, thanks very much ! It enlightens the whole thing for beginners like me. I was thinking to myself: "this guy has an Irish accent", and then I found out you were from Belfast, so my English accent recognition skills are not too bad (I used to live near County Armagh but in the RoI). Greetings from France.
Mamy times i think with one how we can someone cover complete topic with examples, but you proved it that we can cover. Great session and covered complete etl flow, thanks a lot.
I have a notebook that was used to do a job via glue, I need to know how to activate the Job bookmark and how to create a schedule for it. Do you have a video that shows this step by step?
Thank you. Great informational video! I concur with your preference for SQL. Anywhere SQL can be substituted for the bracket and comma usage it is cleaner and less effort.
Hi! Thank you for this amazing video! I have a question: in a glue job I have a dataframe (or equivalently a dynamicframe) with a complex schema that I wrote on my own (using StructType and FieldType available on pyspark). Now I want to create a glue table starting from this dataframe without having to crawl it because I already have my schema defined on this job. How can I create a glue table starting from a dataframe with a defined schema? Is that possible? I thank you in advance for your availability and thank you again for your amazing work :)
Another question I have is: I've noticed that when creating a table in glue it's possible to create a column with type "UNION". Can this be done also in pyspark? I mean creating a dataframe whose schema (defined by me through StructType and FieldType) has a column with two possible types.. I've searched on the internet but I found nothing
Thanks Johnny, great tutorial, when i tried to create notebook, i am getting the following error "Failed to authenticate user due to missing information in request."
![PySpark For AWS Glue Tutorial [FULL COURSE in 100min]](/img/logo.svg){kind=logo}
