How to Schedule & Automatically Run Python Code!

Подписаться 222 тыс.

Просмотров 139 тыс.

50% 1

The first 1000 people to use the link will get a free trial of Skillshare Premium Membership: skl.sh/keithgalli11201
In this video we learn various methods for how we can schedule and automatically run python code. We'll start by looking at cronjobs & windows task scheduler and then we will walk through running tasks in the cloud using AWS Lambda & Cloudwatch. The ultimate goal of this video is to learn how we can automatically generate & send an email with an analytics report every day.
There are methods highlighted in this video that will work on Mac, Linux, & Windows.
~ GitHub resources ~
Link to source code: github.com/KeithGalli/schedul...
Link to analytics report code: github.com/KeithGalli/generat...
~ Relevant videos ~
How to generate an analytics report in python: • How to Generate an Ana...
How to send an email with python (‪@coreyms‬ ): • How to Send Emails Usi...
~ Articles mentioned ~
AWS Lambda memory limits: hackernoon.com/exploring-the-...
Downloading python linux libraries on Windows: / lambda-with-pandas
~ Other resources ~
Python email library docs: docs.python.org/3.8/library/e...
AWS Cron info (scheduling): docs.aws.amazon.com/AmazonClo...
Lambda package setup: docs.aws.amazon.com/lambda/la...
Thank you to Skillshare for sponsoring this video!
-------------------------
Video timeline!
0:00 - What we’ll be doing in this video
0:56 - Check out Skillshare! (sponsored)
1:56 - How can we automate scripts? Overview of local, cloud, and serverless methods
5:18 - Simple example of local script automation w/ cronjobs & windows task scheduler
18:32 - How to schedule code on a cloud machine (use cronjobs)
18:51 - Simple example of cloud script automation w/ AWS Lambda & Cloudwatch
27:09 - Schedule & automate sending an email locally
45:12 - Schedule & automate sending an email on the cloud w/ Lambda & Cloudwatch
50:18 - Installing python packages in a serverless environment (zip uploads)
55:50 - Generate & schedule sending analytics reports (locally)
1:02:45 - Generate & schedule sending analytics reports in AWS Lambda
1:07:03 - Limitations of lambda (max file upload size)
1:09:00 - Generate & schedule sending analytics reports in AWS Lambda
1:18:32 - Final thoughts & video recap!
-------------------------
Follow me on social media!
Instagram | / keithgalli
Twitter | / keithgalli
-------------------------
If you are curious to learn how I make my tutorials, check out this video: • How to Make a High Qua...
Practice your Python Pandas data science skills with problems on StrataScratch!
stratascratch.com/?via=keith
Join the Python Army to get access to perks!
RU-vid - / @keithgalli
Patreon - / keithgalli
*I use affiliate links on the products that I recommend. I may earn a purchase commission or a referral bonus from the usage of these links.

Опубликовано:

3 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 100

@KeithGalli 3 года назад

Hope everyone is doing well! If you have any questions about the material covered, let me know in the comments :) Also, be sure to check out Skillshare: skl.sh/keithgalli11201. The first 1000 people to use the link will get a free trial of Skillshare Premium Membership!

@leonardoduarte7647 3 года назад

Hello Keith Galli, Could you record a quick video explaining how to schedule a script running on the jupyter notebook? You will help me a lot... Thanks, I appreciate your work here on RU-vid.

@woonie3134 3 года назад

Hey could u make a video on food classification and recommendation combined? Classify a meal and recommend recipe plzzz

@legacies9041 3 года назад

I hope you do an advanced course in Keras with custom models and custom compile

@justin3594 3 года назад

I just found your channel a few days ago and it’s already become my favorite python channel. I really love the real world examples. Fantastic content. Please keep it up.

@saulflores5052 3 года назад

This is an amazing resource. I'm using what I am learning from these videos for projects in my internship. Thank you!!!

@krystianmasiak6212 3 года назад

Man, I just want to say thank you. You showed me python, now i'm looking for job in Data Science and i'm much above my friends from studys on that specialization, so... THANK YOU!

@RougeSam 3 года назад

I LITERALLY just needed this solution. Let’s go Keith

@sarathk8183 3 года назад

You are so committed in what you do in all your videos. Thank you so much since I wanted a script I wrote that taps into SQL server -> dB -> Tables -> data frames and dashboard output to run automated....great timing and awesome video

@tyswenson7542 3 года назад

Keith continues to read my mind with video ideas..

@KeithGalli 3 года назад

Lol great minds think alike :)

@aravindashokan Год назад

Exactly

@immanuelsuleiman7550 3 года назад

This is incredibly useful great job as always thank you

@NoahElRhandour 3 года назад

i really appreciate the creative and useful tutorial topics

@EUU100 3 года назад

Thank you for your clear delivery and explanations, loved it. Gave me what i was looking for plus also taught me about other methods and just more general knowledge about python windows and linux. I wish you all the success and happiness you desire

@theholybananaaaandthemiley6815 3 года назад

Amazing Content. I appreciate the effort!!

@itsmesanto Год назад

This is great. Better than paid tutorials out there. Thanks for providing it for free.

@tellulois445 3 года назад

Well done! Bro, Keep making such beautiful Videos

@prasadcaher 3 года назад

Awesome 👍 thanks for making this so simple.🤘

@radabbadaa 3 года назад

Awesome vids from this channel. Keep it up!

@abuaisha93 3 года назад

Thanks for these great videos

@venturibr 3 года назад

This is very nice, thank you!

@andyn6053 3 года назад

Great stuff as always Keith! I have learned more from you than during my whole engineering education at the university!

@eduedu726 3 года назад

thank you !

@juanbetancourt5106 Год назад

Thank you for your knowledge good Sir.

@xxpwnr4lif3xx 3 года назад

I'm new into web-scraping. For school, I'm trying to create a project comparing food prices in in different settings like Chick-Fil-A in stores vs Chick-Fil-A in airports/malls. I watched your other web-scraping videos but I think a video about scraping then converting it over to an excel/csv file would be very nice! Thank you for all the content you do post!!!

@zelaTVV 2 года назад

That's a really good video thank you

@michaeloliveira6745 3 года назад

Awesome! thanks

@ed-salinas-97 3 года назад

This video showed up on my recommendations at the right time. RU-vid reading my mind. I've been trying to wrap my head around Cloud Functions for a while but had not seen any suitable examples until now. Only problem for me, though, is that I want to be able to use Google Cloud Functions, and it's not really clicking for me to translate what you show with AWS Lambda to GCP Cloud Functions. I'm hoping it'll "click" at some point since I'd rather use the GCP Ecosystem. If it doesn't, I'll switch over to AWS Lambda since what you show in this video does seem to make good sense and will make for a good template for my particular projects. Thanks!

@thomasrayanthony4871 3 года назад

WHOAH HERE BEFORE 100k GEETH

@procode6881 3 года назад

Sick video ...really fun one...just a suggestion could you upload videos more often..also i know u are busy with your job

@legacies9041 3 года назад

I hope some day you will do a video on custom models in Keras and other advanced stuff

@cambridgebreaths3581 3 года назад

Keith , how to motivate an absolute beginner to start delving into this giant ocean of programming, honestly feel overwhelmed just opening a book or watching a video, even your amazing videos. THANKS

@vestborobaptistchurch3161 3 года назад

Just do it

@danielrv1325 3 года назад

Hello Cambridge, I would like to give you an advice. Say you want to exercise but there are so many different sports that you don't know where to start... well not doing anything is not a good start, see what fits you better or what you like and then try-out. Indeed programming is a "giant ocean", so first you should know what area/s interests you. Select a broad area (for example web development or data science) and then go deeper or even change/combine paths as you advance. Motivation is second tier, what you need is being constant and resilient. You don't build a house on a motivated day, each day you place some bricks and after sometime, you have a house. Sure, some days you may be more motivated so you advance faster, but being constant means that even in a day you are not motivated at-all you still make progress and that is key for getting closer to your goal. Being resilient helps since (I assume you don't come from a programming background, neither did I) some concepts might get complicated so a strong mindset is needed to continue learning and not quitting or waiting again till you feel motivated.

@cambridgebreaths3581 3 года назад

@@danielrv1325 THANK YOU KINDLY DANIEL! REALLY GREAT ADVICE. THANKS AGAIN.

@vestborobaptistchurch3161 3 года назад

@@cambridgebreaths3581 wow where’s my thank you? Let me just say that, even though I spent a lot of time with sites like freecodecamp and sololearn, I didn’t really become what I would consider to be a semi competent programmer until I had to do it for my job. I was inheriting a lot of python scripts from a boss who left and that’s when I realized I could use it to make stuff that was really cool and practical. I used the scripts he built to get a basic and practical foundation, and then I went to Keith’s videos to expand my skills. There’s just all sorts of cool shit you can do with python that is practically useful. For instance, I used my basic knowledge to pull covid data from an api and analyze it, which is something I used to do with Excel but that was so slow and lame. I can also use api requests to make tables of marijuana strains and their genetics, which is something I’m interested in. There are so many practical guides on RU-vid to build a project from start to finish: just copying one of those line by line will build your skills and all you have to do is modify a few things to make wholly different projects. It’s not about just gaining knowledge for its own sake but making practically useful things. You’re lucky to be diving into this during a time when a lot of the tutorials are ready made and there is a huge knowledge base on sites like stack exchange that will answer most questions you have. So just do it. Build something

@cambridgebreaths3581 3 года назад

@@vestborobaptistchurch3161 Thanks Vestbord. This one programming advice, to just do it and start programming, (overlooked inadvertently!), summaries the best programming advices.Thanks again

@venn5318 3 года назад

Man I love your vids you are the best! Its hard to get a nice answer because everyone calls themselves professionals these days but you are the gold. So here goes my question: Is there any real great book/books or courses you can suggest for python oop + algorithms in general? Also a personal question >> which programming languages you think I should crash? In school we started with python but I find c++ so fun as well.. or what other languages I should master for getting a good job and being well skilled? Really hoping you answer me.Thank you 😭🥺

@vincentthoma9889 3 года назад

Hi, can you make a video about typical questions, which a data scientist has to answer?

@anilv9788 3 года назад

Awesome your 🙏

@huat99hotmail Год назад

great

@patrickdamour1728 3 года назад

Cute dog!

@KeithGalli 3 года назад

😊

@fentonmsu 3 года назад

I could use that to set up links to pages to share my RU-vid videos

@jamestang1737 3 года назад

Thanks for another practical walkthrough, Keith. Question here-I'm on a windows and trying to set up task scheduler with a python script. Running into an "no module named selenium" (example as thats one of the modules in my script) but I had set both python and anaconda to my path. Also tried a batch file where I first list the python.exe path along with my script but that also doesnt work. Any obvious pointers? Thanks in advance man

@thomasrayanthony4871 3 года назад

Hey Keith! You should do a hair tutorial! Get a bowl cut or mullet!

@python360 3 года назад

Nice video. I leave a Pi 4 running 24/7 for CRON jobs and web scraping. Ps why did you use port 465 for smtp and not 587 ? I thought 465 was deprecated.

@BJCP 3 года назад

Hey Keith, would it be possible to cover making a python program into a service? I have some monitoring tasks that I'd like to run that continue to monitor files in a folder. It would be really nice to make this into service / background task.

@Qibbles 3 года назад

I use a raspberry PI zero to run my scripts and cronjob. Hardly any overhead and I can leave it 24/7!

@KeithGalli 3 года назад

Great combo!

@milesmena4994 Год назад

I'm here because I want to do a similar task to texting yourself a picture of a dog!

@debjyotibanerjee7750 3 года назад

We don't need to install python specially for Kali app, if we already have it for windows, right?

@stevegabrial1106 3 года назад

Hi, is any playlist like Data Science or Machine learning in 100 days. Thanks

@BiologyIsHot 2 года назад

Surprised this didn't include the Python module "schedule!"

@rajith9562 8 месяцев назад

Hey Keith! Amazing video man...totally love it! I was trying out the Dog photo messaging which you just showed in it and I was getting only the url of the photo gotten from the api instead of the photo when the message pops up in my phone. Can you help me with it!? Thanks

@cozyvibes6178 Год назад

19:40 Best way to host and run phython script for free with AWS

@TheRavi24 3 месяца назад

Hi Keith please could you please show how to run in azure pipeline ? 😢

@jthomassaldias Год назад

Hello Keith, I have followed your instructions to schedule a run script from phyton but Im unable to find out what is wrong... the scheduler simply does not run the script... i have tried with .exe files and do but not the .py.... running the .py outside the shell performs as expected. What would you recommend? Thank you. Thomas

@DanielMartinez-cx9bp 3 года назад

Do a Kaggle comp.

@MrTheyosyos 3 года назад

1) r'c:\full\path' == 'c:/full/path' 2) To get a full path og a file: Hold shift + right mouse click on file -> "copy as path"

@KeithGalli 3 года назад

Useful tips, thanks! :)

@MrTheyosyos 3 года назад

@@KeithGalli No, Thank you.

@pbottiglione 2 года назад

Hi Keith quick question. To schedule a Jupyter notebook job do I need to save it as a python py format first and then use the window scheduler? or can I do it on the notebook directly using the import schedule function? If I use the import schedule function say I run it on virtual machine do I need to keep it open in my browser in order to work? Thank Keith

@sgkg7178 2 года назад

Were you able to find a solution for this?

@tushardeepsingh4976 Год назад

Keith I want to schedule my python code every week at a particular time . Request you to please suggest what to do

@KeithGalli Год назад

I recommend hosting a cheap machine in the cloud and running a cron job at a weekly interval. I think this is the simplest to setup, reliable solution

@vasantkumar5539 2 года назад

Can you create guide on ApScheduler python package

@matorac Год назад

Apscheduler is great for windows, linux or mac. I created web app using flask+apscheduler,+angular for running py jobs, viewing logs.. everey job has its own py env using pipenv.

@daji1952 7 месяцев назад

@KeithGalli thanks for the video but i am having a problem I am having python file and i did the same crontab as shown in video i am not ale to run the file automatically i can see cron job it is proper but now i am not able to understand what went wrong [not from coding background]

@thomasrayanthony4871 3 года назад

Me and my 7 siblings are your subscribers!

@KeithGalli 3 года назад

Thank you!!!

@TheSergiGarcia 3 года назад

Hi Keith! I keep having an error message on AWS Lambda like this 'SMTPAuthenticationError'. It looks like I can't login into my gmail, but it works if I run the code in the terminal. I also enabled access to less secure apps in gmail, but still it doesn't work. I was wondering if you have had this problem before or not.

@noahh0578 11 месяцев назад

Hey Keith. I am running into an inssue using the scheduling method via AWS. I have python code that flips through 3 phone numbers, but when I schedule it to run every day at 11am my code is executed from the starting phone number. And instead of having 1 Log stream with each event every day at 11am a new logstream is created everyday. Why might this be happening?

@deltreseshabattawowowwwow6825 3 года назад

Rah rah rah! I like muffins!

@dookshi 3 года назад

Great tips! What about running Python code on Android phones? There are schedulers but we're missing the first and foremost a Python interpreter here. :-( Given all the great Python content on your channel (so simple yet so powerful) how come there's nothing about running Python on the one platform out there capable of continuously generating huge amounts of invaluable data from all the sensors that smartphones have? Running ML models on real-time data in our pockets! Aren't Python+ML and smartphone sensors a match made in heaven???

@AlexMercersRevenge Год назад

Whenever I make a automated python script I keep running into Windows User Account Control popup. Python can't do ANYTHING from here. How do I use python to work around this so I can make a useful python script?

@raymondchan3587 2 года назад

Install and Import schedule is much easier.

@apricotsareyummycrunchandg1872 3 года назад

Apricot

@kevingrondin974 3 года назад

Do you use wsl 1 or wsl 2 ?

@KeithGalli 3 года назад

I think I use wsl 2 at this point, but to be honest I'm not even sure. Don't think the version should matter for the purpose of this video.

@jargon8610 3 года назад

Hey Keith i need help my Sublime text 3 always errors it always shows this: Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases. [Finished in 0.5s] If any of the viewers know this you can help fix it

@TengoT 2 года назад

Hey, can you please help me to schedule a job in python every 2 hours starting at some particular time. I wrote the code as - schedule.every(2).hours.at("01:00").do(func). but this is not working.

@KeithGalli 2 года назад

I recommend using one of the methods I described in this video. Hard to keep a python script actively executing for extended periods of time. Also if you do the method you described, you'll probably want to make sure the time zone is correct, because by default it will probably be UTC.

@danielrv1325 3 года назад

When I run the crontab -l or -e, I get the following text: 'Support/E3FE50D5-DB55-470F-8B7A-2B574AC40711/43139812-B903-4DD3-84F0-395806C84045 h >/dev/null 2>&1' I've already gave full disk access to terminal but still haven't find anything on stackoverflow or other site to make crontab -l display the same. Unsure what this is :(

@KeithGalli 3 года назад

What OS are you on and what terminal are you using?

@danielrv1325 3 года назад

@@KeithGalli OS Catalina version 10.15.7 and the Mac terminal version 2.10

@jpdkhuntiya8540 3 года назад

How to graphic design in Python tools

@armandoacevedoluna3393 3 года назад

yoyoyoyoyo yo! hahahaha

@huili1920 Год назад

error:

@NomadicBrian 3 года назад

I don't like WSL to start with. Too many pieces of Microsoft tools required. Not worth it if I have to use that.

@EUU100 3 года назад

any other suggestions?

@NomadicBrian 3 года назад

@@EUU100 Sorry WSL is a trigger. I plunged into Linux Mint because I had Windows 10 crash for good on my Dell Inspiron. I'm comfortable with Debian packages and what I can run in docker images. Today I'm running Python in Jupyter Notebook against a MySQL database over a virtual network that maps client port to container port. I'd have to review the podcast here on schedulers and get back to you. If you can split boot right into Linux I'd do that over virtual Linux on Windows 10.

@thomasrayanthony4871 3 года назад

Would you like to partner with my company for merch