Тёмный

BULK Write Back to SQL from Power BI 

BI Elite
Подписаться 70 тыс.
Просмотров 32 тыс.
50% 1

In this Power BI tutorial, you'll learn how to write back to a SQL database from Power BI! With this technique you can connect to any data source and insert all of the rows into a database table.
To enroll in my introductory or advanced Power BI courses:
training.bielite.com/
Elite Power BI Consulting:
bielite.com/
Data Insights Tools:
www.impktful.com/
Connect with me on Twitter!
/ powerbielite
0:00 Intro
1:06 Basic INSERT
8:26 Only Inserting Unique Values
10:24 Final Demo

Наука

Опубликовано:

 

5 июл 2024

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 99   
@shyamchandankalgutkar4720
@shyamchandankalgutkar4720 2 года назад
Its a great video and hats off to you for sharing this. In case there are multiple sources just make sure to set the Privacy settings to same such as Private or Organizational in both the sources. Many Thanks.
@ramyagopinath8320
@ramyagopinath8320 2 года назад
Perfect video that helped me write back to postgresql. I published my pbix and tried a incremental refresh so that every week I write back only new records to postgresql table which isn't writing back but the refresh is successful and its showing up new modified date in report
@mre_man
@mre_man 2 года назад
A simple and very intuitive solution. It was easy for me to implement! thanks a lot and much appreciated.
@rajarshisingh2547
@rajarshisingh2547 2 года назад
I have been looking to something as such to be done from power bi itself. My firm doesn't allow me to use powerapps for writeback to database. You just gave me a solution which will be very beneficial to me.
@irfanshaikh262
@irfanshaikh262 2 года назад
Its so amazing that when you need to find something related to MSBI and you get it on the BI Elite channel. Thank you so much for lending me this logic I've been struggling to find. Love from India❤❤❤
@BIElite
@BIElite 2 года назад
Haha glad to hear it, Ifran!
@Erebosaither
@Erebosaither 2 года назад
This is really great! Congrats and you have a new sub :)
@julescaruso4398
@julescaruso4398 2 года назад
Excellent as always! Thanks!
@MuhammadBerki
@MuhammadBerki 2 года назад
Great video Parker 👍
2 года назад
Great writeback! Thanks!
@Grace-fz4og
@Grace-fz4og Год назад
You save my day! thank you
@kennynguyennz
@kennynguyennz 2 года назад
That's really cool. Thanks for sharing.
@cristian.angyal
@cristian.angyal 2 года назад
WOW!!! That's a really awesome idea to write the INSERT Statement in a Custom Column and use a Custom Function to run it. Great work Parker! Thanks for sharing and stay assured I will totally "steal" this and use it in my Daily work ! Thanks again!
@BIElite
@BIElite 2 года назад
No problem! Yes, please use this!
@cristian.angyal
@cristian.angyal 2 года назад
@@BIElite I've just tested it getting data from a SharePoint list with Power Query in Excel and it totally works!!! This opens a lot of use-cases !!! Thanks again for sharing!
@BIElite
@BIElite 2 года назад
Love it, Cristian! Thanks for sharing your use case.
@PatrickWallet
@PatrickWallet 2 года назад
This can be very usefull! Thank you for this tutorial.
@BIElite
@BIElite 2 года назад
No problem!
@jawwadchaudhry367
@jawwadchaudhry367 2 года назад
This is really awesome!!
@tejasbrahmbatt5344
@tejasbrahmbatt5344 2 года назад
As always great video.. will it be possible to write data based on DAX conditions.. Let say the user interacts with the report and changes some slicers etc.. Then the final table | results should be written back ? Thanks
@rakeshnag70
@rakeshnag70 2 года назад
Thanks for sharing. It is really helpful
@BIElite
@BIElite 2 года назад
Glad to hear, Rakesh!
@ErikaLoomis
@ErikaLoomis 6 месяцев назад
This is fantastic, thank you!
@abhishekstatus_7
@abhishekstatus_7 2 года назад
Excellent Solution as always by Parker ! Thanks a ton ! Your solutions keep us going back to check to implement something new in our report. Also would like to know if you are planning to have Power Apps Integration with Power BI, Financial reporting or NLP kind of implementation with the help of Power Apps and Power BI on RU-vid! Thanks a ton again:)
@christophertauss3640
@christophertauss3640 Год назад
Excellent demo. I am looking at something like this to insert records into an Access db housed in SharePoint. One concern I have with this approach is that there is no way (I am aware off) to write the record asynchronously. So if multiple users try to insert records at the same time, sometimes Access has real problems. But I am continually amazed by what Power Query can do. I would estimate that at least 50% of VBA code can now be automated with Power Query. I have subscribed and I look forward to your other videos.
@AI-Health-posts
@AI-Health-posts 2 года назад
It’s really awesome technique
@bachirouadama5826
@bachirouadama5826 2 года назад
Very very interesting. Well done
@BIElite
@BIElite 2 года назад
Thanks!
@dansilecchia7355
@dansilecchia7355 4 месяца назад
It took a little re-jiggering for my purposes, but I got it to work. If you want to replace the data, you would some how do a drop table function at some point. Trying to figure that part out now. Thanks!
@haribhaktapujari4763
@haribhaktapujari4763 2 года назад
Very useful. Thank you
@BIElite
@BIElite 2 года назад
No problem!
@harshitdoshi2004
@harshitdoshi2004 2 года назад
This is an excellent solution. We just have to mindful that we don't start using this method as semi ETL :)
@majdyazigi8185
@majdyazigi8185 2 года назад
Great one, thank you, does it work with MS Access database?
@muhammadtanveerislam5998
@muhammadtanveerislam5998 2 года назад
Sir it is really good tutorial.
@BIElite
@BIElite 2 года назад
Thanks!
@ajaaskelainen
@ajaaskelainen 2 года назад
Something very new :) thnk you!
@tomeering9228
@tomeering9228 2 года назад
Hi Parker, Thank you verry much for making this video. I have copied your code and when i publish it to PowerBi Service it is not able to refresh, is that something on my end or do you experiance the same? In the Desktop app it works great !
@jamesmundy3127
@jamesmundy3127 Год назад
Parker - Great video. I applied to a local MySQL database and it worked like a charm. I had to modify my table a bit to create a unique constraint but everything is working fine now. Do you think this same approach would work with a snowflake database? Seems like a database server is a server and an insert statement is an insert statement. Might be a good weekend project. Thanks again.
@Fabiano1223
@Fabiano1223 Год назад
Astonishing !!!!!!
@SamundraDarion
@SamundraDarion Год назад
This has been BRILLIANT! I do have one request though - you created an ID column to load - but you don't cover the creation of that column, I'm trying to piece loose things together now.
@jorgedelgado4846
@jorgedelgado4846 2 года назад
Hi Parker, then we can publish and configure scheduled updates so that the process continues in Power Bi Service? or is only done on power bi desktop?
@mad.z9064
@mad.z9064 2 года назад
It’s really awesome, and thank you for sharing! Can we write measures and calculated column values with this trick?
@ThankYouGoogle4
@ThankYouGoogle4 2 года назад
Logically you won't be able to write measures back using this method because it's modelled calculation. Any column you wanna write back to the SQL must be available in the querying data while applying this logic. You can invoke custom column while transforming your query and then add it in the function column created. That should work.
@infobymattcole7724
@infobymattcole7724 Год назад
Wow. Well done.
@kevinwallst
@kevinwallst 9 месяцев назад
Thanks for sharing the great video! Would this require the premium license of PowerBI?
@lekhasri2360
@lekhasri2360 2 года назад
This really helps a lot..I am actually searching and struggling for this option to write back to SQL where I have billions of records to be written back to SQL from my Power BI report
@Flowerloveesun
@Flowerloveesun 2 года назад
Hi Parker, many thanks for sharing this excellent video. Just one question, would there be any efficiency issue if the size of the table is so big like 50000+ rows?
@nagendraveera1577
@nagendraveera1577 Год назад
use ADF :)
@trevorshao
@trevorshao 2 года назад
Hi i follow your step and script exactly but some how the message column doesnt show: "INSERT INTO FromView (rownum,Category) VALUES ( '" & Text.From([rownum]) & "', '" & Text.From([Category]) & "', ); SELECT 'Inserted Row' AS [Message];"
@iancockcroft9427
@iancockcroft9427 Год назад
Great video. thanks. gives me some ideas to work on. this isnt true write back? its just moving data from one source to another. is it possible for users to write a comment on the report and write that back. without power apps? I use a pagenated report with an insert query and pass parameters to it, but surely there is an easier way?
@VladMZ
@VladMZ 2 года назад
This is an interesting solution to my challenge to store historical copies of the data produced inside Power Query. I do have a problem though with initiating a new SQL session for every insert. It's not truly "bulk", and may be quite slow with a decent number of rows. I will try to move it outside of the function, and may be pass the session object as a parameter.
@BIElite
@BIElite 2 года назад
That’s a very good point… please let me know what you come up with.
@rajarshisingh2547
@rajarshisingh2547 2 года назад
Good thought, I shall also try the same
@michaelc2712
@michaelc2712 2 года назад
Parker, nice video to explain this. For many rows wouldn't it be more efficient to create a stored procedure in SQL and pass the rows as one JSON value, chunking into a reasonable number of rows?
@samaguire1
@samaguire1 2 года назад
yip - way more efficent
@Miketochy
@Miketochy 2 года назад
Kindly share what the procedure would look like Thanks
@laplasparadox3138
@laplasparadox3138 Год назад
Any follow up on this
@moindharwad1130
@moindharwad1130 2 года назад
@BI Lite I tried the above solution but at the end am getting please rebuild the data combination error.. At it is exactly pointing at invoked custom function step in the power query Bi Lite please guide us from here.. Thanks
@bethkemp6934
@bethkemp6934 2 года назад
Can this be done using Power BI Report Server, or is this for the cloud/service only?
@davidpimental6704
@davidpimental6704 Год назад
Is it possible to do updates in a similar way? What about Bulk Updates - such as every record with the same issue?
@jimihieu
@jimihieu 2 года назад
Amazing technique! How do we add another column as insert timestamp as well please? And is it better to create that column in power query then insert to sql or use sql function while inserting?
@ThankYouGoogle4
@ThankYouGoogle4 2 года назад
Add one more column for timestamp in the SQL format you like (I prefer UNIX timestamp)... He's basically passing a SQL query, and it's always best to have insert timestamp at place where you're actually inserting data (this makes sure there isn't any discrepancy in case of insert failure for a specific row).
@jimihieu
@jimihieu 2 года назад
@@ThankYouGoogle4 im not good in sql, can you please help to expand the code please?
@ThankYouGoogle4
@ThankYouGoogle4 2 года назад
@@jimihieu add a custom column with hardcoded value for all rows as- UNIX_TIMESTAMP() Then just add the column in the m query to pass it just like other 3 column values are passed in insert statement. Use - "&Text.From([column_name])&"
@mre_man
@mre_man 2 года назад
Are you using MS SQL? What I did was create a date column datetime not null then set the default Value or Binding to (CONVERT([datetime],datediff(day,(0),getdate()))) in your SQL table . Whenever you add a new row it will automatically stamp the date for you. No need to pass or stamp the date manually through PowerBI or SQL.
@AnkitKshatriya
@AnkitKshatriya 2 года назад
Can this be done using Excel PowerQuery?
@ShivShankar-kw1uc
@ShivShankar-kw1uc 14 дней назад
Kindly reply, if we need to replace data with fresh records however it must stay unique, what I need to write as a code.
@mohaarulez
@mohaarulez 2 года назад
What abour sql injection? Does power bi or forms handle that behind the scenes?
@BIElite
@BIElite 2 года назад
Very good point. No, it runs the “native query” against your database, so anything goes.
@padreloco9733
@padreloco9733 2 года назад
Thank you again for an amazing video, useful. But I need more guidance to overcome a challenge I am facing: my Power BI is connected to a file that is adjourned every day. In other words, each file is replaced by a new file each day. The structure doesn't change of course. My question is: since I don't have those indexes to keep track of which row to replace or not, is there a way to make sure that I have replaced all old data in the "old" table with new data that just came? Thank you again.
@ThankYouGoogle4
@ThankYouGoogle4 2 года назад
If you are receiving files of same structure. Create an SFTP (legacy or custom) location and put original file there. Create a shell script to rename any incoming file to the original file's name and then replace the file. This way you'll always have a full load in Power BI. In case you can create an index which is highly recommended (maybe try using a composite key) to determine the changes, then go for incremental refresh on the composite key.
@padreloco9733
@padreloco9733 2 года назад
@@ThankYouGoogle4 thank you for your answer. Theoretically this answer seems to me feasible. I have to try it in practice though. Thanks again.
@padreloco9733
@padreloco9733 2 года назад
@@ThankYouGoogle4 Remaining in Power Query, during fiction creation, what if a code is added there with two purposes: 1- Drop the existing table (with data to replace, if it exists) 2- Create another table in the database and lastly load data...?
@ThankYouGoogle4
@ThankYouGoogle4 2 года назад
@@padreloco9733 I don't think i fully understood the requirement here because from what i understood is that you want to drop the data, and then load the new data. But that what a full load (refresh) does. Can you create your question on power bi community with your requirement and I'll be able to create a sample pbi file for you.
@AmitKumar-ox1qk
@AmitKumar-ox1qk 5 месяцев назад
can this type of query work with MariaDB too?
@mirrrvelll5164
@mirrrvelll5164 11 месяцев назад
Good one, can you also delete from sql too? I assume I need to delete some rows too... p.s your page BIElite is quite bad responsive....I cant access it easily.
@sameerulhaq4066
@sameerulhaq4066 5 дней назад
how to make it automatically detect the Item code and system time and datae and user email. I just want them to add comemnts and they cant edit the date time, email or item code? I have tables having relationship based on Item code?
@tangtom2478
@tangtom2478 2 года назад
I guess this solution is also working on Excel, if it is, we can design a template in Excel, and ask user to input the data, then eventually save the data into data lake.
@diaconescutiberiu7535
@diaconescutiberiu7535 Год назад
After I write the insert statement and click ok .. i get null for about 90% of my rows. The odd part is ... that it worked for about 10% !? Any ideas?
@sameerulhaq4066
@sameerulhaq4066 5 дней назад
but how will it take input from front end if I give to the user?
@ncbshiva
@ncbshiva 2 года назад
Is it possible to publish this pbix file and schedule a refresh in power bi service ?
@BIElite
@BIElite 2 года назад
Yes! That’s one of the great things about this. Scheduled refreshes will write data to the db on a schedule 8 times per day with pro.
@didijaba
@didijaba 2 года назад
@@BIElite can you show that also on video?
@jackstrudley1039
@jackstrudley1039 Год назад
@@BIElite Great video. In Desktop it's perfect but when I attempt it in the Power BI Service nothing happens (the refresh is successful but no write back occurs and I receive no error messages). Can you show this working?
@thing72
@thing72 Год назад
it doesnt seems to write new rows when data set is refreshed thru gateway :( or its just me? when i open PBI report and refresh manualy, it makes records into DB
@ncbshiva
@ncbshiva 2 года назад
Can you provide the PBIX file used for this demo ?
@BIElite
@BIElite 2 года назад
The PBIX would be very personal to my data and SQL table, but I will try to post something. Might not be too helpful though.
@snehalgawari6756
@snehalgawari6756 Год назад
But from which form we are writing back?
@gregamos7987
@gregamos7987 Год назад
Trying to take this process and modify it to write data to an sqLite .db via an ODBC connection. It works OK until the final step (Invoke Custom Function), where it fails with this message: Expression.Error: Native queries aren't supported by this value. Any ideas?
@thosewhoslaytogetherstayto3412
Same issue here
@remib5863
@remib5863 2 года назад
I like the idea for the possibilities it brings in very specifics use cases but Ouch! I can already see Cyber Security teams pulling their hair out and having a heart attack with this kind of tricks! Hopefully people will understand what they are doing otherwise it might become messy. Power Automate might be a "cleaner" alternative to move data, not sure if bulk upload exists with this either.
@AmitKumar-ox1qk
@AmitKumar-ox1qk 5 месяцев назад
Hi there, I am getting following error - An error occurred in the ‘’ query. Expression.Error: Native queries aren't supported by this value. Details: [Table]
@anuradhavummaneni7372
@anuradhavummaneni7372 Год назад
how to get pbix file of this
@druthorah
@druthorah 2 года назад
⭐⭐⭐⭐⭐
@mudassirsyedrashidali9787
@mudassirsyedrashidali9787 2 года назад
I can't see Chris Wagner in the comments
@BIElite
@BIElite 2 года назад
Should we be expecting Chris?
@trynewthings5927
@trynewthings5927 Год назад
getting an error. I don't think you can do this anymore
@kaisquared90
@kaisquared90 2 года назад
You know what else starts with "SQL In-" ...
Далее
Power BI - Write Data to SQL
9:49
Просмотров 60 тыс.
Sending Data from Power BI to SQL
5:02
Просмотров 11 тыс.
Power BI: Write back using Power Apps!
17:17
Просмотров 4,7 тыс.
Choose a phone for your mom
0:20
Просмотров 5 млн