Intro to Python Dask: Easy Big Data Analytics with Pandas! 

YW. Thanks for watching.

It may make it easier depending on what you are trying to do. Seems like keeping the image attributes with the image might make it easier to use both together.

Yeah. I mean you could use pandas with SQL but that would not scale. You could create a view in the SQL database and query that with read_sql_table. It is a limitation. I think Spark wins on that one.

@@BryanCafferky Thanks. I do have another question maybe related to DASK or jupyter notebook. So, the data I am currently working on has over 50MM rows. It takes a long time to read in via pyodbc even though I read in chuncks. I then converted to dask dataframe, hoping it would fasten the data manipulations. I called .persist() after adding 3 more columns, it would run for 30 minutes until I get a memory error. I am using my company desktop (96GB available, 12 precessors), so I'm surprised it's taking that much memory. Groupby seems to take a long time as well. I tried to use R data.table table, and hasn't had a memory error yet. Do you happen to encounter the same situation before, or do you have any guess on what could be the problem causing this issue?

@@jamiew3986 If possible, maybe create a table or view on the source database that limits the columns and data you need. Also, limit large columns like substring to get part of a string, etc. The read this in and directly write it to a parquet file. Once that has been done. Load the parquet file. Not sure why its running out of memory but you don't have scale out. This would be easier on a cloud environment. Here are some links that may help. docs.dask.org/en/stable/dataframe-sql.html Measuring memory usage docs.dask.org/en/stable/generated/dask.dataframe.DataFrame.memory_usage.html A Blog Use Case with Detailed Info on Optimization blog.dask.org/2021/03/11/dask_memory_usage#:~:text=When%20possible%2C%20you%20can%20fine,wasting%20RAM%20or%20CPU%20cores. Hope that helps.

When your dataframes are taking a significant chunk of available memory, it's good to think of trying Dask. You do get an error though. See this blog. towardsdatascience.com/how-to-avoid-memory-errors-with-pandas-22366e1371b1

@@BryanCafferky Thanks Bryan!

Yes. pandas is a Python library so this video is not useful if you don't know Python pandas.

@@BryanCafferky I guess I need to finish python course first 😅

@@I677000 Focus on pandas. You don't need to become an expert on all of Python. The book Python for Data Analysis by Wes McKinney is a good one.

Hmmm..Ok. Sorry to hear that. For Azure SQL, Databricks might be a better option.

@@BryanCafferky would love to use databricks, but system is far to mature to change it. Contacted Microsoft and appears my issue has to do with the version of Alquemy…. Only works with version 1.4 and below.

Dask is more like Apache Spark, i.e., scales out. Polar is a data science library.