What is an ETL tool, why is it important to have and how do you decide which is best for your organization? Case Study: www.intricity.... Talk with a Specialist: www.intricity.... www.intricity.com
I recommend you watch the video regarding Data Virtualization, as it lightly hits on this topic. You want to avoid embedding data integration logic in multiple tools as these tools don't share this logic with competing ETL vendors. This means organizations end up with a lot of rework, simply because they have more than one tool.
Nice presentation. However ETL tools in general are limited to only a certain type of data. At the begining of the videos it is shown that chairs are represented by data and some attributes/parameters are shown. While those are needed and do not fully represent the product (here the chair). There are all sort of other type of data needed thst ETL tools can't deal with at all. To name one your CAD data. Your CAD data will drive the creation of many data downstream the ETL tool will use. As far as a discrete manufacturer is concerned ETL tools can help but it usage can only be limited to certain type of data which often are not even the source. This raises new questions. Where should you manage your data? At the source or at some uncontrolled level downstream. One key thing is to integrate your CAD with a database and once in you can export. However I am yet to see a so called ETL tool to work with a PDM/PLM system. I would be glad to read the views of the different experts out there on this issue. Thx
It is true that unstructured sources can introduce a challenge to many ETL tools. However, the ETL tools at the top right of the Gartner Quadrant have Unstructured data options, as well as Hadoop based integration which is an excellent target for unstructured data.
Intricity101 Thank you for your quick reply. I would be careful in saying that CAD are unstructured data. The engineering community will not like it. CAD as anything can also be very structured and properly managed. Especially when integrated in a PDM/PLM system. This is where I think a ETL tools could "connect" as a PDM/PLM system relies on a database, eg Windchill PDMLink is usually used with Oracle. The world of data is huge and to my knowledge there is no system which does it all. In addition data does not belong to a system, they belong to the business. The key is to ensure consistency across the systems as data has to smoothly flow across systems and be there at the right state at the right time when a user needs it. To me the major benefit is instead of adding new system to try to link things together, to simplify the system landscape. So if a company has 4 ERP; they should migrate to 1, if they have 4 PLM they should migrate to 1 etc... While this is costly on a short term compare to putting additional systems in between to try linking, it is definitively better on the long term. My thoughts :)
Ja Ka Unstructured data is data without a row and column relationship. Think of text files, pdf files, and media files. You might for example receive a regular PDF file which represents an order which you want to read automatically. Because the PDF file has no defined schema you have to parse the data to ingest it into a row and column format.
NacNac Mott If the data is structured into a relational database like Oracle, then it is definitely accessible to an ETL tool. However, access does not spell out feasibility. Sometimes the application data is at such a low level that piecing together the logic for how a user experiences it is just too much work. Often application vendors will release an API to simplify the querying into these systems. However, the ETL tools must be able to call those APIs natively to leverage them. Most ETL vendors only select a few highly adopted applications to build native API access. (like big ERP and CRM applications)
Soooo many. Informatica, SSIS, Data Stage, Talend, Matillion, SAP Data Integrator, Snaplogic, AWS Glue, Alooma, Fivetran, and Stitch. That's not all of them just a sample of a bunch. To see some of the important updates I recommend you check out the ETL vs ELT video. (recent one)
I was excited to hear about glue when it was announced. I was very surprised to learn about its back end (hadoop). Strange to me that they wouldn't just use their own native capabilities in S3 and EC2.
