SSIS Row Limits and DefaultBufferMaxRows

I had an odd issue during some testing I was doing on a custom source component I’m writing for EBCDIC data conversion.  I first assumed that surely my code was at fault but the test just seemed to simple to be “my code”.  So here’s the scenario,  I have a file with 100,000 bytes of the number ‘1′ in EBCDIC.  I created a simple package to import the file as 1 byte rows into a Dataset Destination.  I was surprised when I executed the package and found that only 65,536 rows were processed.  Now if you have been around software for more than a week you probably recognize this magic number and unfortunately it triggers all sorts of theories about what could possibly be going wrong.  Well, I’m still trying to investigate the “why” and I’ll update this post and welcome comments if anyone has input but for now the solution to this oddity is to increase the DefaultBufferMaxRows to greater than your number of records.  This seems like a horrible workaround for a situation that should happen quite often so the hunt continues…