The context is that I had a request to add a feature to a Nodejs app that I wrote a couple of years ago. The purpose of the app is to grab a bunch of files off of an SFTP server and make files out of them. The purpose of this post is to provide a google friendly summary so nobody has to spend as much time on this as I did.
So, I fired up the app on my development machine. It looked ok at first but then it just stopped processing. Further investigation told me that the downloaded files got to disk and that the problem was that the ‘close’ event was never firing.
Inspiration strikes and I realize that the app was written for an older version of Nodejs. More experimentation revealed that the app runs correctly on versions up to 9.11.2 but at 10.0.0, it fails. For google, I restate: upgrade nodejs to version ten causes readable stream to never fire ‘close’ (or ‘end’ or any other) event.
After a full day and a half of research and experimentation, I learned that, starting in nodejs 10, readable streams default to ‘pause’ mode. Previously, they were in ‘flowing’ mode. In ‘pause’, the stream waits for the consumer to take data before it does anything else.
Well, sort of. It does provide the data, somehow. After all, my zip files were created and did work correctly.
In any case, I found a clue someplace that there had been a change to the default for streams. I found another clue that talked about the ‘paused’ behavior. Eventually I found a comment on stackoverflow (here) that mentioned read();
So, here’s the code:
sftpControls.sftpConnection
.get(remoteDirectoryPath + name, false, null)
.then(zipSourceStream => {
const filePath = `${zipFileDirPath}/${name}`;
zipSourceStream
.pipe(qtools.fs.createWriteStream(filePath, { encoding: null }))
.on('close', () => {
zipSourceStream.destroy();
next('', name);
});
zipSourceStream.read(); //tell the stream to be ‘flowing’
})
That is the entire change. Add one read() and the stream becomes ‘flowing’ and emits a close event when it’s done.
A small warning. I tried to chain the .read() along with the pipe() and on() methods. That failed. The read() method does not return the source object. You will get an error, “no read().pipe()”. Reverse it and get “no pipe().read()”. You have to run it like it’s shown, directly off of the original stream object.
If anyone reading this actually understands why this is happening or how it works, I am sure future programmers will be grateful to read the comments. I know I will.