I have a multiple dim tables which are about 50 million,40m, 80 million records in it , I'm trying to load their respective surrogate keys into a fact table . To get the surrogate keys I'm using Lookup transformation with config as full cache, Fail component and lookup match output. The problem here is there more millions of records to lookup and lookup transformation requires lot of memory. Is there any other better way to get the surrogate keys to be loaded into the fact table?
Any help is appreciated!!
Thanks,
sree