Creating And Transforming Streams With Node.js

What Are Streams In Node.js?

Streams in Node.js are objects that let you read data from a source or write data to a destination in continuous fashion. They are great for saving on memory consumption, as the data is quite literally "streamed" in bite sized chunks. This still might be a little confusing, so let me walk you through an example of using streams.

A Very Large Text File

I have created a 120MB file that is filled with alternating a and b characters.

abababababababababababababababababababababababababababababababab
abababababababababababababababababababababababababababababababab
abababababababababababababababababababababababababababababababab
abababababababababababababababababababababababababababababababab
abababababababababababababababababababababababababababababababab
abababababababababababababababababababababababababababababababab
...more

You get the picture. Now what if I wanted to open this file in my Node.js code, and write it's contents to another file? There are two ways you could go about this.

Utilizing FS.ReadFile And FS.WriteFile

In this scenario, we will use the fs.readFile and fs.writeFile methods. Reading the data in this fashion means that we will load the entire text file into memory before we write it's contents to our new text file.

const fs = require('fs');

fs.readFile('./data.txt', { encoding: 'utf-8' }, (err, data) => {
  if (err) {
    console.log(err);
  }

  fs.writeFile('./data-two.txt', data, () => {
    console.log('written');
  });
});

This would cause our script to skyrocket to 120MB's worth of memory consumption. There is a better way to do this.

Utilizing Node.js Streams

In order to rectify our memory consumption problem, we will make use of a readable and writeable stream.

const createWriteStream = require('fs').createWriteStream;
const createReadStream = require('fs').createReadStream;

const read = createReadStream('./data.txt', {
  encoding: 'utf-8',
});

const writer = createWriteStream('./data-two.txt');

read.pipe(writer);

Not only is this much better on memory consumption, the code also looks cleaner because it doesn't utilize any nested callback functions.

Transforming The Stream

If streams weren't already cool enough, you can also transform them. For this example, let's create a transform that converts every a into a c, and every b into a d. All that's required is that we create a new transform, and utilize the pipe method to run through our transform before it's written to the text file.

const createWriteStream = require('fs').createWriteStream;
const createReadStream = require('fs').createReadStream;
const Transform = require('stream').Transform;

const read = createReadStream('./data.txt', {
  encoding: 'utf-8',
});

const transform = new Transform();

transform._transform = (chunk, _, done) => {
  let str = chunk.toString();
  str = str.replace(/a/g, 'c');
  str = str.replace(/b/g, 'd');
  done(null, str);
};

const writer = createWriteStream('./data-two.txt');

read.pipe(transform).pipe(writer);

As you can see, the transform is placed in between the reader and writer with the pipe method. This way we can replace the a's and b's chunk by chunk, not the entire file at once.

Summary Of Node.js Streams

That's about it for Node.js streams for now folks. If you have any problems understanding any of this, feel free to shoot me a message, and I'll make sure it's crystal clear.