Тёмный
No video :(

AWS Kinesis Tutorial for Beginners [FULL COURSE in 65 mins] 

Johnny Chivers
Подписаться 21 тыс.
Просмотров 63 тыс.
50% 1

ℹ️ johnnychivers....
☕www.buymeacoff...
📁 github.com/joh...
☕ www.buymeacoff...
00:09 - What the course will cover
00:54 - Set Up Work
05:43 - Kinesis Streams Theory
09:01 - SDK Vs KPL Theory
10:31 - Kinesis Data Streams Practical
12:03 - Kinesis SDK
15:54 - KPL Practical
22:26 - Lambda Consumer Theory
23:19 - Lambda Consumer Practical
31:45 - KCL Theory
32:43 - KCL Practical
37:24 - Studio Note Book Theory
38:19 - Studio Note Book Practical
44:22 - Kinesis Firehose Theory
45:17 - Kinesis Firehose Practical
53:27 - Kinesis Analytics Theory
54:03 - Kinesis Analytics Practical
In this course we take a look at AWS Kinesis in full using the console. In under 65mins we go from zero to hero. The course covers the AWS Kinesis Data Streams, AWS Kinesis Data Analytics and AWS Kinesis Firehose. Over the course of 65mins we create a fully functioning kinesis streaming application.
😎 About me
I have spent the last decade being immersed in the world of big data working as a consultant for some the globe's biggest companies.My journey into the world of data was not the most conventional. I started my career working as performance analyst in professional sport at the top level's of both rugby and football. I then transitioned into a career in data and computing. This journey culminated in the study of a Masters degree in Software development. Alongside many a professional certification in AWS and MS SQL Server.

Опубликовано:

 

5 сен 2024

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 58   
@xeroxeroxeroxeroxeroxeroxero
Things to clean up when you're done: - Delete the CloudFormation Template - Empty and Delete the S3 Buckets housing your data - Delete the lambda function created in lab 6
@picassoofai4061
@picassoofai4061 3 месяца назад
yeah, AWS billed me 300$. I use AWS SAM/Terraform now.
@sany2k8
@sany2k8 2 года назад
This guy is awesome 😎, deserves more subscribers. Keep up your good work on data engineering. I am a full time software engineer planning to move on data engineering.
@JohnnyChivers
@JohnnyChivers 2 года назад
Thanks for watching the channel Sany!
@shoooozzzz
@shoooozzzz Год назад
This video took me from Kinesis zero to hero! Thanks, mate. I now feel comfortable producing and consuming data from Kinesis
@OttoTtt
@OttoTtt Год назад
Guys, don't forget to shut down everything on Zeppelin (Johnny says about this in the video). I missed one query and got a $16 bill after one day. Better delete the whole Cloud Formation right after you finish this video.
@DakFour
@DakFour 9 месяцев назад
I deleted the cloudformation template an hour later after finishing the course. i dont remember shutting down Zeppelin. i got a bill of $35.In Billing , Kinesis analytics, Opensearch and ec2 were active( but nowhere to delete them in the console and CLI) it did stop after 24h which meant some applications were still sending data. How would i prevent that in the future. i thought deleting the cloudformation would do the job
@akshayhs2766
@akshayhs2766 2 года назад
Hi Johnny. Your AWS Glue video was super helpful. Happy to be one of the first viewers of this video !!!
@JohnnyChivers
@JohnnyChivers 2 года назад
Glad the AWS Glue video was useful - and thanks for the on going support.
@book_beats
@book_beats 2 года назад
Hello from Ukraine, Johnny! Thanks for your videos! At this point i`ve watched all videos on aws glue and kinesis, they are really great.
@JohnnyChivers
@JohnnyChivers 2 года назад
Glad you like them!
@andrewaylward7654
@andrewaylward7654 6 месяцев назад
Love the Belfast accent! Good work Johnny!
@jeffgoodrich278
@jeffgoodrich278 8 месяцев назад
You're awesome Johnny C! I'm transitioning into data engineering in the next 6 months and your videos are great! I would like to study for the AWS solutions architect certs, and I will pay you for a custom project and learning plan. Let me know if you would be open for that, I am an engineer by training, so I do not need to be spoon fed everything, but I do want to take the most effective and efficient path. Thank you!
@Chuukwudi
@Chuukwudi Год назад
Why did you not use Kinesis Data Generator ? Why did you write your own custom data generator? Is the data generator presently broken? I have been trying for the last 48 hours to get KDG to work but without success.
@LeeLeeCode
@LeeLeeCode Год назад
Thank you so much!! Very nice and detailed instructions. Much appreciated!
@vishnupandey3390
@vishnupandey3390 Месяц назад
19:56 mvn clean compile package will not work we need to change pom.xml file with the following text
@skviknesh
@skviknesh Год назад
Thanks so much! This was awesome!!!! Do more....
@rakeshmali1727
@rakeshmali1727 8 месяцев назад
one question: do we need to mention which shard a consumer can listen/read to? or the customer automatically discovers the shard to read from?
@atulkumar7030
@atulkumar7030 Год назад
Awesome Going! Thanks for all your hard work in putting up these videos (y)
@shayankabasi160
@shayankabasi160 Год назад
Hi Johnny - Thanks for the nice tutorial, could you please check provide some work-through of the yml script that you are using in cloudformation section for resource creation.
@Chuukwudi
@Chuukwudi Год назад
I wanted to request the same!
@milogodo100pre
@milogodo100pre 2 месяца назад
hI, I've been trying to do an exercise which consist on ingest data from an website (currencies), store them, and then show in a graphic the data collected, that's very simple to say but very difficult for me to do it, do you have any information I will really appreciate it. I have the API key from the source of data
@yinwong667
@yinwong667 2 года назад
Thank you for your awesome tutorial! But can you also teach us how to clean up?
@TheSuperJCN
@TheSuperJCN 19 дней назад
Whenever I launch the CF template it fails with this error: Resource handler returned message: "ServiceExecutionRole has insufficient permission to perform glue:GetDatabase. (Service: KinesisAnalyticsV2, Status Code: 400, Request ID: 0543627e-528b-453e-8e23-9781a5525d9b) ServiceExecutionRole is a role within the template and it does have permissions for glue:GetDatabase (as these are specified in a policy also created with the template) Haven't been able to get around this error to do the tutorial. Has anyone faced this issue? Note: I have Lake Formation enabled in the region I am doing the tutorial, but the error does not exactly mention missing LF permissions, but an IAM action, plusI did try using broad LF permissions but still got the same error
@vipulkumar6082
@vipulkumar6082 Год назад
Bookmark 48:00
@barisdnmz
@barisdnmz Год назад
nice tutorial, by the way how to clean up the environment that all set up?
@databloke2755
@databloke2755 10 месяцев назад
I dont see the jar file in kinesis-producer-library-examples-master folder
@magneat
@magneat Год назад
thanks a lot!
@leamon9024
@leamon9024 Год назад
Hi, is it possible to enable "dynamic partitioning" in kinesis firehose when integrating with cloudwatch subscription filter? I tried it, but it reports an error saying "Only UTF-8 encoded data is supported for dynamic partitioning records.". So now my workaround is Cloudwatch -> subscription filter -> lambda -> kinesis firehose -> S3, which I'm not sure if it's a best practice.
@erickjhormanromero6905
@erickjhormanromero6905 Год назад
KPL Practical is throwing the next exception Could not copy native binaries to temp directory
@jessemcmullen-crummey3158
@jessemcmullen-crummey3158 Год назад
was going great until the maven compilation failed for me around 20:00. Got an error: " Fatal error compiling: java.lang.ExceptionInInitializerError: Unable to make field private com.sun.tools.javac.processing.JavacPro..." :(
@JohnnyChivers
@JohnnyChivers Год назад
Hi Jesse there’s been changes since I did the video. You need to update the POM file to have the latest version of Lombok and that should solve the problem.
@geoffreyli552
@geoffreyli552 Год назад
btw, fixed the 2nd POM issue by changing the java version to lower one, by #sudo update-alternatives --config java and #sudo update-alternatives --config javac.
@ayaanquadri5203
@ayaanquadri5203 Год назад
same.
@vishnupandey3390
@vishnupandey3390 Месяц назад
change pom.xml file with the following text
@parikshithshivaprakash5523
@parikshithshivaprakash5523 2 года назад
how to do streaming for large files
@lingxiankong4791
@lingxiankong4791 2 года назад
Thank you for guiding this learning!
@JohnnyChivers
@JohnnyChivers 2 года назад
Thanks for watching!
@smokecracy5831
@smokecracy5831 Год назад
bravo
@undesinyvalley
@undesinyvalley Год назад
thank you Jack black
@srinivastadinada3571
@srinivastadinada3571 2 года назад
Gem!
@JohnnyChivers
@JohnnyChivers 2 года назад
Thanks for watching.
@anshikagupta4931
@anshikagupta4931 Год назад
niceee
@arupde6320
@arupde6320 2 года назад
be regular
@bruh_1283
@bruh_1283 2 месяца назад
Is this legit
@xeroxeroxeroxeroxeroxeroxero
So far, great tutorial. I wanted to point out the Java does not compile. No big deal, as this part can be skipped because the stream already has data on it from the previous step [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project amazon-kinesis-replay: Fatal error compiling: java.lang.ExceptionInInitializerError: Unable to make field private com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors com.sun.tools.javac.processing.JavacProcessingEnvironment.discoveredProcs accessible: module jdk.compiler does not "opens com.sun.tools.javac.processing" to unnamed module @6b350309 -> [Help 1] Sorry, I don't know Java. Otherwise I'd make a PR to fix it.
@christian112
@christian112 Год назад
I managed to get it running with these updates to the pom.xml file: Old:************************* [...] 8 2.15.66 0.14.6 [...] New: ************************* [...] 11 2.19.33 0.15.7 [...] Edit: Later, for the kcl-app, you will need to update the amazon-kinesis-client version to 2.4.8 to avoid a similar error: software.amazon.kinesis amazon-kinesis-client 2.4.8
@vipul7010
@vipul7010 Год назад
@@christian112 thanks it works!
@kaizen9554
@kaizen9554 Год назад
@@christian112 very helpful. thanks
@aaronvillegas7900
@aaronvillegas7900 Год назад
@@christian112 This worked! Thank you!
@madhurnayak55
@madhurnayak55 8 месяцев назад
@@christian112 Thanks for helping out.
@nitinsaini05
@nitinsaini05 2 года назад
Thanks for your video & efforts!! YAML file kinesis-immersion-day-cfn.yaml errored while uploading to create Stack. Something got changed over the period ?
@JohnnyChivers
@JohnnyChivers 2 года назад
Hi Nitin, I just managed to create the stack successfully in the North Virgina region with my Admin user. What region did you run it in? It took 20mins to create in full.
@nageshwarburman8819
@nageshwarburman8819 Год назад
I have been getting this error when I ran "mvn clean compile package" : Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project amazon-kinesis-replay: Fatal error compiling: java.lang.ExceptionInInitializerError: Unable to make field private com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors com.sun.tools.javac.processing.JavacProcessingEnvironment.discoveredProcs accessible: module jdk.compiler does not "opens com.sun.tools.javac.processing" to unnamed module @6b350309 -> [Help 1] Did something change by the time when this tutorial was created to now?
@parakhtayal1254
@parakhtayal1254 Год назад
same, now in progress to solve that :(
@halloweex
@halloweex 8 месяцев назад
same
Далее
Cute kitty gadget 💛💕
00:23
Просмотров 4 млн
ЭТО мне КУПИЛИ ПОДПИСЧИКИ 📦
22:33
AWS EMR Tutorial [FULL COURSE in 60mins]
1:01:06
Просмотров 61 тыс.
AWS Glue ETL Vs EMR - Which one should I use?
8:05
Просмотров 39 тыс.
Kubernetes 101 workshop - complete hands-on
3:56:03
Просмотров 1,6 млн
Cute kitty gadget 💛💕
00:23
Просмотров 4 млн