Developing a SQL Server Integration Services Custom Data Source Data Flow Component.
SSIS has a maximum buffer size and row limit to "chunk" work based on buffer size.
This requires a source component to recognize when the buffer is full, and then suspend until another buffer is available and resume loading from the source to the new buffer.
What I don't see covered anywhere is:
- how to effectively deal with a full buffer
> calculate the size, count rows and ensure closing the buffer before overflow (except that I can't see how to get the max size or max rows from the parent data flow task)
> wait for a buffer overflow exception and rewind to the last row/bytes written to resume with a new buffer.
- how to store the "bytes/rows to skip" information for the component to pick up the next time it's invoked by SSIS.
SSIS has a maximum buffer size and row limit to "chunk" work based on buffer size.
This requires a source component to recognize when the buffer is full, and then suspend until another buffer is available and resume loading from the source to the new buffer.
What I don't see covered anywhere is:
- how to effectively deal with a full buffer
> calculate the size, count rows and ensure closing the buffer before overflow (except that I can't see how to get the max size or max rows from the parent data flow task)
> wait for a buffer overflow exception and rewind to the last row/bytes written to resume with a new buffer.
- how to store the "bytes/rows to skip" information for the component to pick up the next time it's invoked by SSIS.
Neil Thomson
