Hey everyone,
I'm going to admit upfront, I'm pretty new to the SSIS world, I have background in SQL Server but not much with these DW tools.
Here is my "situation"
I have a tool that when used creates a new database per project (uses MySQL). Each project has a shared schema, however depending on which plugins within the tool are run will determine whether a specific table exists or not. I'm in the early stages of beginning to build a DW for these projects. We have used this tool for some time but don't have a good way to perform historical analysis on the data as each project is fragmented into its own database and there is little data cleansing performed.
Here is my initial desire/"vision"
I was going to create an SSIS package that performs the initial ELT on these databases, what I would like would be to create a situation where I could do as follows:
Connects to our MySQL database server where there are multiple databases per project/user. I assume I can use a solution similar to here, here and here. All of these databases share a similar schema, however they may not all have the same "extra" tables (all have the same core tables).
Loops through and in turn connects to each database and first imports the core tables that are the same for every table and puts them in a staging database.
Then, and this is the part I'm not sure on (I think I can figure out the stuff leading up to this). I'd like it to import all "extra tables that exist in the current database" ie all extra tables are prefaced with the same leading name like"preface_table_name" so I would like it to import all tables it finds with that preface.
I'm comfortable with stored procs and the like so I planned on utilizing them to then dice and clean up the various imported tables for adding them to the dw.
I'm more then willing to put in the work figuring this out, but was hoping someone could point me in the right direction. It's a bit of a crazy project for my first serious work with SSIS and I'm trying to wrap my head around all the concepts. If I am completely out in left field, by all means correct me as needed.
Thanks in advance for any help you can lend!