I want to increase the internal buffer value of node.js

○Problems

I am writing a proxy with node.js.
Proxying heavy POST content caused disconnection.

クライアント Client → ode node.js Proxy → Web Web Server

As for the proxy method, I am sending the chunk I received at the data event at の to the web server little by little.
With this method, node.js might not be able to withstand it in the middle, or it will disconnect if heavy content is POSTed.Light content POST works.

By the way, once you receive all the POST data from the client and send the request to the web server, it works well.
The only thing that doesn't work is the process of receiving data events and sending them to the web server repeatedly.

After a lot of research,
The node.js stream has a buffer called highWaterMark inside, and I think that the total amount has reached its limit.

Is there a way to increase this highWaterMark?

The Http server is created in the following manner:

https.createServer(options, function(request, response){

If we can increase the highWaterMark for this request and response, we think it will be solved.

I understand that you can increase highWaterMark with the constructor of steram, but is it possible to increase it with the generated object like the one above?

I can't post all the source code for a reason, but
Basically, if you don't put too much pressure on it, it works correctly and comfortably, so you can answer on the premise that there is nothing wrong with the program.

Supplemental

I forgot to write it, but when I looked at the official website of node.js, it said that if I connect it with PIPE, I don't have to worry about the internal buffer, but even if I connect it with PIPE, I still get an error due to the load.
So I wondered if I could increase the internal buffer.

node.js

2022-09-29 21:33

1 Answers

I've never tried it, but

How to increase highWaterMark

is
https://techblog.yahoo.co.jp/advent-calendar-2016/node-stream-highwatermark/
Is helpful?

The above uses fs.createReadStream, but it seems that highWaterMark has been adjusted.

Below is an additional note.

I looked it up and found that I could do it in the following way.

'use strict';

constfs=require('fs');
const https=require('https');

https.createServer({
  key —fs.readFileSync ('./certs/server.key'),
  cert:fs.readFileSync('./certs/server.crt')
}, (request, response) = > {
  request._readableState.highWaterMark = 16*1024*2;
  console.log(request._readableState.highWaterMark);
  response.writeHead(200,{
    'Content-Type': 'text/plain'
  });
  response.write('Hello World');
  response.end();
})
.listen(8888);

As for the method of investigation, I looked at it as follows.

HighWaterMark seemed to be used by extracting this._state.highWaterMark value from Readable.prototype.read, so if you could rewrite this._state.highWaterMark, it would have been possible to use https.createServer.
https://github.com/nodejs/node/blob/7dfeb36143a37b5f11766220a6408b9e6124a53b/lib/_stream_readable.js#L355
https://github.com/nodejs/node/blob/7dfeb36143a37b5f11766220a6408b9e6124a53b/lib/_stream_readable.js#L429

This._state.highWaterMark was set to ReadableState when the Readable of the _stream_readable module was new, and it seemed that it could be rewritten if it could eventually access Readable._state.highWaterMark.
https://github.com/nodejs/node/blob/7dfeb36143a37b5f11766220a6408b9e6124a53b/lib/_stream_readable.js#L131
https://github.com/nodejs/node/blob/7dfeb36143a37b5f11766220a6408b9e6124a53b/lib/_stream_readable.js#L71-L76

The ClientRequest received in the requestListener of createServer inherited OutgoingMessage, and OutgoingMessage seemed to inherit Stream, so I decided if _state.highWaterMark of ClientRequest could be rewritten, so I wrote the above sample code.
https://github.com/nodejs/node/blob/7dfeb36143a37b5f11766220a6408b9e6124a53b/lib/_http_client.js#L77

2022-09-29 21:33

If you have any answers or tips

Popular Tags

python x 4647

android x 1593

java x 1494

javascript x 1427

c x 927

c++ x 878

ruby-on-rails x 696

php x 692

python3 x 685

html x 656