I’ve spent a week or so at this point working on a rendezvous problem based on my understanding of the course content from the Berkeley class that I mentioned in an earlier post.
I’ve got something that successfully streams, and successfully allows data to rendezvous, and successfully writes out the data at the end.
Small data sets that I can hand-check are working perfectly, but something is going wrong with larger data sets and I’m getting outputs saying that more numbers were processed than were in my data set.
It seems to be an issue with the code that writes the unmatched data from the hash table out to the unmatched data file. I’d like to continue working through it, but need to get back to the consulting work that has started stacking up, so I’m going to have to set this to one side.
However, I’m going to link to the code because I’m worried that if I don’t, that I’ll never post the result of so much time and effort.
A key takeaway for me on the project (in addition to all of the learning about how streams work in NodeJS), is that when all was said and done, it turns out that there is an “EventStream” npm module that would have probably cut a whole bunch of time out of my efforts.
I’m seeing a recurring theme in my development efforts where I end up trying to re-invent the wheel because I don’t know what resources are out there.
Some of that will probably improve through the simple passage of time, but part of my educational efforts in the future will include running through npm packages so that I in the future I’ll have an idea of what resources exist to speed my efforts up.
Also, here is a link to the post that pointed me towards the EventStream npm module:
https://itnext.io/using-node-js-to-read-really-really-large-files-pt-1-d2057fe76b33