Combine CSV files using Python

Подписаться 64 тыс.

Просмотров 44 тыс.

50% 1

Manually combining CSV files into one master is time consuming, and labor intensive, and especially if you have a large number of CSV files. So why not write a Python program to automate the task? And the best part is it will take less than 5 minutes to write the code.
In this tutorial, we will learn how to create a Python program to automate CSV files merging.
Buy Me a Coffee? Your support is much appreciated!
----------------------------------------------------------------------------------------------------------------
PayPal Me: www.paypal.me/...
Venmo: @Jie-Jenn
Support my channel so I can continue making free contents
---------------------------------------------------------------------------------------------------------------
Patreon: / jiejenn
By shopping on Amazon → amzn.to/2JkGeMD
More tutorial videos on my website → LearnDataAnaly...
Follow Me on Facebook → / learn-data-analysis-10...
Business Inquiring: RU-vid@LearnDataAnalysis.org
Tag:
#PandasTutorial #DataAnalysis or

Опубликовано:

27 сен 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 66

@BiswarupMandal Год назад

This is really helpful. What I liked about this code is that it is very short and simple.

@chetansonar9320 3 года назад

You have no idea how much this videos has helped me... Amazing! I am gonna hit subscribe and press the bell icon to that... Cheers!

@paranoya733 Год назад

After bullshiting my job interview, i found this video.. thank u

@nurulfadillah1248 Год назад

very useful. this is the only tutorial that gets me the right result. thank you very much

@amirtahaehsaei1609 3 года назад

it hit right at the spot, many thanks

@austinhomolka 3 года назад

What a great video. Thank you for making it. It's amazing how powerful of a tool you can create from just a few lines of code. Thanks for the tutorial Jie!

@mariannygodoymacias2957 3 года назад

just what I needed, thank you so much :D

@tatendagutuza3866 Год назад

Good stuff. How do I adjust the code when the csv files are located in a different location with the python file?

@rajvardhansinghpanwar3921 6 месяцев назад

Mine is showing dataframe has no attribute 'append'

@zhichaozhang1580 3 года назад

thanks for your courses

@pankajgoikar4158 2 года назад

Thank you. Saved my time.

@michael3226 2 года назад

I have an error or NO COLUMN TO PARSE FROM FILE. How can I handle this?

@Krauzerin11 6 месяцев назад

Great video, just real confusing your miss pronunciation of the word "object", it sounds like Agile, which gets you confused

@user-fw3vv4id5z Год назад

Thank you, very helpful!!

@tokelo2547 Год назад

Mine is not saving the combined files but the script is not returning an error

@lovrosibenik1070 Год назад

How do I specify the folder from where I want to merge my files

@saimali_2001 3 года назад

thanks a lot for making this!

@toxiclife 2 года назад

Hey, great video, please do tell, if I want to merge all CSV files into one workbook but with different sheets only, like the combined workbook will have 3 sheets separately only. How to do that.

@swarnavdeb2064 2 года назад

you didnt mentioned the specified path

@jiejenn 2 года назад

Not sure if I understand your question.

@swarnavdeb2064 2 года назад

@@jiejenn What I meant was inorder to read a file a specific file path is required, but in this case the file path is not given so where exactly is the data being retrieved from?

@jiejenn 2 года назад

Please Google os.getcwd().

@gam3rman85 9 месяцев назад

why are the files called AAPL if they are TSLA stock?

@RahulCKapatkar Месяц назад

Becauseeeee !!!! ITTTT ITT ITTT DOESN'T MATTER WHAT YOU THINK IT IS !!!!!!!!!!!!!!!!! Cause Stone COLD SAID SO !!!!!

@drujene 2 года назад

this helps a lot, and i have a question. what if there is non english in that csv files, how do you construct it to english?

@jiejenn 2 года назад

You probably need to translate the content first.

@Dennis_Smits 2 года назад

line 7 if file.endswith('.csv') ^ SyntaxError: expected ':' Process finished with exit code 1 This is de error code i get, is the any one who can help

@jiejenn 2 года назад

You are missing a colon at the end.

@brunodobelinguimaraes4900 3 года назад

really good video my friend

@azizt6773 6 месяцев назад

i believe there are been a change to the script for the DataFrame for this line master_df = master_df.append(pd.read_csv(file)) ^^^^^^^^^^^^^^^^^^^ the word append doesn't get recognized since the system throws an exception that says AttributeError: 'DataFrame' object has no attribute 'append'. Did you mean: '_append'? EDIT!!: on the line i have mentioned u need to add ._append instead of .append only just like this master_df = master_df._append(pd.read_csv(file)) just copy what i did then change the variable name to your desire

@jiejenn 6 месяцев назад

I just tried, it is working for me. But if that still doesn't work for you, try master_df = pd.concat([master_df, pd.read_csv(file)], ignore_index=True)

@johnpineda7692 Год назад

Thanks Bro!

@mitragupta7548 3 года назад

What if i want to sum their data into 1 file

@nguyenduyta7136 2 года назад

Hi thank for your great videos. I have a question, if I have many csv file in one folder ( such as a list stock files A.csv, B.csv,…..) and now I would like to take one column of each files ( such as “close price” column) and finally concatenate all of them to one file. So how can I do

@jiejenn 2 года назад

You can combine all the files first into a master, then delete the columns you don't want.

@nguyenduyta7136 2 года назад

@@jiejenn hi Thank for your reply, by the way, I have many csv file A.csv, B.csv, C.csv, D.csv, E.csv,…. in one folder, but how can I use “for loop “ to pick automatically specified one ( B, C, D) only but not all of them ?

@mohammeddanishreza4902 Год назад

@@nguyenduyta7136 I think you can put the needed files name in a list and loop over the list.

@shomugger 2 года назад

did the same code in VS code and got a lot of errors, where might be a problem?

@jiejenn 2 года назад

Not sure.

@sportsmania2841 2 года назад

Hi , I have two row headers so it skips one row but second row keeps on repeating. I dont want the headers to repeat only data to be concatenate under the first file header lines. can you help?

@JAZZMAINAWABBS 2 года назад

did you get your answer ? am also stuck

@sportsmania2841 2 года назад

@@JAZZMAINAWABBS not yet, still waiting.....😐

@harithameka8328 2 года назад

pandas.errors.EmptyDataError: No columns to parse from file i am getting this error how can i solve it sir

@jiejenn 2 года назад

Please Google the error for answers.

@asmitasubedi8215 3 года назад

What can I do to combine CSV files by column and not by row? I want more than 300 files combined side by side. I would appreciate it if you could answer. Thanks!

@jiejenn 3 года назад

Look into pandas concat method.

@shabs_jovial 3 года назад

Can you please put a video on how to save multiple dataframe to a single zip using pandas without saving as CSV

@jiejenn 3 года назад

I don't think you can save a dataset as a zip file.

@shabs_jovial 3 года назад

@@jiejenn well we can by using compression='gzip' while we calling to_csv() .... But I could not find a feasible way to compress multiple df in a single zip

@alexybogomolov 3 года назад

@@shabs_jovial why don't you want to create the csv file first? Create it in other directory, use shutil.make_archive(output_filename, 'zip', dirname) to write the zip file, then delete the csv file.

@shabs_jovial 3 года назад

@@alexybogomolov Thanks for tip. But I have limited space and have abundant RAM and my result dataframe is more than a GB to save as CSV. Likewise i have multiple scripts running on this condition. So when I tried to save as CSV and run , my other scripts fail due to the restrictions of my space

@inaammuaz2284 3 года назад

I dunno why, but mine is duplicated

@jiejenn 3 года назад

I dunno why either.

@whyleesworld2190 3 года назад

Can you make this py file into a executable file? or use it as universal csv consolidator?

@jiejenn 3 года назад

You can make a Python program an executable file with pyinstaller.

@asimjavaid5836 Год назад

Your Subscription and hit like is really due on every learner.... Great teaches

@benmcdermott8575 2 года назад

Do you know how to solve the issue of no such file or directory. Even though it does exist?

@jiejenn 2 года назад

Yes. The script cannot find the file.

@Abhishekkumar-bl4kl 2 года назад

hello sir, please make a video for .sav file combine using python

@jiejenn 2 года назад

Hum... Not interested. Thank you.

@pankajgoikar4158 Год назад

Thank you again.

@KhalilYasser 3 года назад

Awesome as usual Jie Jenn.

@Mia-td3bx 2 года назад

love it!!!

@antisunny9226 2 года назад

How to get N number of csv files as user input ?

@jiejenn 2 года назад

Not sure if I understand your question.

@antisunny9226 2 года назад

@@jiejenn HI, Thanks For immediate response. My query is i had some csv files , I need to take some of them as user input and combine them Into one single file and the perform some EDA. Like some times 3 files as user input Some files 5 files as user input.