NodeJS

Lets say you are implementing a REST endpoint that lists a collection. Normally, youd paginate the endpoint with a maximum page size of say 25 items to avoid memory issues. Youd return something like this example, and if the client wants more than one page, it would need to make more than one request: { "data": [ { "id": 1, "title": "Item 1" }, { "... id": 2, "title": "Item 2" } ], "pages": { "current": 2, "prev": 1, "hasPrev": true, "next": 3, "hasNext": true, "total": 5 }, "items": { "perPage": 2, "begin": 3, "end": 4, "total": 9 } } This can be a real pain for clients to deal with, especially if the client knows it wants everything. With NodeJS, you may be able to greatly increase the maximum page size or eliminate it completely with streams. This is because with streams you dont need to buffer the entire response before sending it. You can send the response piece by piece, greatly reducing the memory demands on the server. If you are using MongoDB, you can get a streaming response from the database like this: This will be an objectMode stream, with each object one document from your MongoDB collection. HapiJS also supports streaming responses like this: HapiJS response streams should be in binary mode or in objectMode emitting strings. You cant just connect your MongoDB stream to HapiJS because you need to construct a valid JSON object so the client can parse it. To do that, you can pipe your MongoDB stream through a Transform stream that takes in MongoDB documents and emits a JSON object piece by piece that will look like the response above. Heres an example of a stream that does that: // PaginationStream.js const Transform = require('stream').Transform; const assert = require('assert'); class PaginationStream extends Transform { /** * Create a PaginationStream * @param page {Number} which page of data will be streamed through (starting with 1) * @param perPage {Number} how many objects are returned per page (= 0; if 0, then return all objects) * @param total {Number} the total number of results (= 0) */ ...Read More