Hi Folks, I am facing a dilemma, Could someone please assist.
In this project there will be multiple sources which may have different architecture and different type too. So far I have encountered only one source <Excel>. It contains 35 columns which have different kinds of data types. So what I did is that I created an ODS having temp and staging table. In the temp table all my columns are NULL and VARCHAR, So that any abnormal row in source also can be loaded in this table. Now on this table I am doing all kinds of transformations needed. After all the transformation I am loading the data into various dimensions and fact tables which I have created for the project.
This was so far, Now the problem or dilemma I am facing is that when other sources will come I may have to change my dimensions and fact tables. But that's okay. Below is the design diagram which I am trying to build.
<Picture is not getting uploaded, will try again and post it>
My manager is putting stress on having just One Temp table at first level and One Staging Table at second level. It shall have all the columns coming from different source and then all the Transformations should happen in this and then load into Dimensions and Facts.
So what should I do. I shall go by what I am thinking or my manager's approach?
Chaos isn’t a pit. Chaos is a ladder. Many who try to climb it fail and never get to try again. The fall breaks them. And some are given a chance to climb, but they refuse. They cling to the realm, or the gods, or love. Illusions. Only the ladder is real. The climb is all there is.