I am working on a data integration project. The requirement is as follows
1) Call a rest api(GET) to get all the organizations GET /organizations
2) For eachorganization get all the networks GET /organizations/[organizationId]/networks
3) Foreach network get all the devices GET /networks/[networkId]/devices
Store all this to data in DB and create and XML file for each organization with its related network and devices.
I know how to achieve this sequentially using Script Component in SSIS. During the initial stages this wont be a problem but as the data grows performance will be an issue. I have limited knowledge when it comes to parallel/asynchronous programming.
Can some please help me with an efficient design solution with some samplepseudocode.
Ideally I would want to do this using SSIS but I open for any suggestions(for ex a console app)