Much better :) gg

@@codeacademia sorry, actually that was for 10000 objects per batch, for 1000 is around 4s. I wonder if we could do better. Which side represents a bottleneck - nodejs or mongodb inside the container. Also, I am running this on macbook pro with 32GB of memory and around 2Gs of memory for the container. I ran the mongostat inside the container to get some stats - I got around 31,073 inserts/sec.

31k per second is actually pretty descent for 1 cpu core. I can say there is room for improvement if you still haven't reached 100% of your CPU. You can split your data into multiple pieces and run multiple nodejs instances, each one reading from the data and inserting in MongoDb... At some point, you will need to scale your db and make it optimized for heavy write tasks as well!

Good catch, default highwatermark in nodejs streams is indeed 16Kb. You can double-check the documentation for that. The number of pipes influences the highwatermark as well as other factors, I'm not aware of it all, but i know from a fact that the size can change, and i let nodejs take care of it and adjust the speed by itself... There is no good/clear documentation about streams in general, so feel free to experiment things :) and make sure you share it with us :)

What angular version are you using? I've given you the github repository so you can take it as an example.

Glad you liked it. Enjoy 😉

The master and slave are always in sync. The same data is shared between the slaves and the master. If the Master goes down, Sentienl switches the role of any slave he finds to become Master. If the Master that went down becomes up again, Sentienel will switch its role to Slave.

@@codeacademia cheers for the answer, one issue here is, there is always a permission error, when sentinel trys to rewrite the config, not sure i you get that too, with the latest redis, i used your config to do the test but yeah in the sentinel container there is permission issues

@@codeacademiaso basiclly it doesn't switch the slave to master if i take down the master

Permission issues! What kind of permission? Could you share the issue?

the fail over defiantly don't work with your configs @@codeacademia

Thank you so much 😀

Man thanks 😊

Sorry about that

My pleasure, part 2 and 3 have more interesting stuff as well :) Best of luck

Its on github, check the description section.

Could you check in the console if there is any error? Hard to guess without a hint

@@codeacademia Hey.. Thanks for such quick response. There is actually no error at powershell, also I checked browser console there is none while landing on port but as soon I try to hit some endpoints manually (posts, login) in URL then error appears in console.

I see, what errors are you seeing? What angular version r u using?

@@codeacademia Hi..here is the list of packages and versions I'm using - - > Angular - 15.2.8 Angular CLI - 15.2.6 Node - 18.14.2 NPM - 9.5.1 OS - win32 x64 Angular flex-layout - 15.0.0-beta.42 (deprecated) [ I'm not quite sure if the error is related to this package being deprecated ] The error I'm getting on browser console is - - > NG0203: inject() must be called from an injection context.

That's the issue, I'm using angular 12 :( Can't help with that

Share the issue so i can help you

@@codeacademia Thank you bro i was solve it one of issue is the redis version and second one is permisssion but i was solve them

Thanks :)

Easy :) Instead of ndjson formatter that I've created, create another one for csv, and use createWriteStream to save it in a file. That's it.

@@codeacademia But it doesn't save the data in the desired format it saves the whole object in csv file

You should change the format by yourself. Use Something like Object.values(myObject).join(';') to create a csv, instead of objects use strings ... I can send a code that does that, but you need to learn a bit, experiment things :)

@@codeacademia Thnks i got the solution btw great series on node streams

Glad to hear that, please, like or share to support the channel 🙏

Thanks for reaching out. Make sure that your last transform doesn't send any data. I've explained that in this tutorial, the last transform should behave like a write stream, its consums the data but doesn't send any.

Could you explain what you want to do? I don't understand what you want to do with PassThrough?

@@codeacademia So my use case is reading remote file, transforming it and return it as readable stream, PassThrough or Transform can do that and does fine with small file. If i do not send any data returned stream is empty Here is example async function transformData() { const sourceFileStream = fs.createReadStream('fileName.json'); const resultStream = new PassThrough({ objectMode: true, transform(value, enc, cb) { cb(null); }, }); await pipeline(sourceFileStream, resultStream); return resultStream; } The problem is that original example works fine with small file but timesout on bigger one :S Originally i was using sourceFileStream.pipe(resultStream) And everything worked well

That has nothing to do the size of the file. Streams work great with huge files. For your use case use Transform not a PassThrough. And make sure to follow the requirements Await pipeline( yourReadStream, yourTransform, lastTransform) Make sure that LastTransform doesn't send any data in your callback. as explained in the videos :) Best of luck.

@@codeacademia If anyone comes in with this question answer is highWaterMark, because node streams has protection and is build for piping information, it makes sense to have some limit of read data which is highWaterMark. In my case i needed to load all buffer in to the memory, which means i need to have high highWaterMark, which will impact application memory used but lets me pass streams around. Not suggesting to do this to anyone as for my case its bad architecture and legacy code :(

I'm not aware of any pipeline status that can be applied.

@@codeacademia It would be helpful, especially for large files to check the progress of the stream.

Agreed, you can use a transform to keep track of "consumed" chunked divided by the total of chuck (objects or csv lines ...)

Check avro-js

Glad you liked it :)

@@codeacademia 100% :)

CsvToJson takes care of it, yes. And yes, the transform will continue to work without problems :) I suggest leaving hightWaterMark in its default value.

@@codeacademia how could it continue to work if the chunk you receive represents half of an object or more than one object.. because your transform function expects a single object as input and a single object as output. No?

Under the hood, csvToJson will make sure to send you an object. If it receives less or more, it will make sure to send you a valid object down the pipeline...

@@codeacademia Yeah i discovered that using fetch i have to JSON.parse(chunk.toString())

Ndjson file is a simple file that contains multiple json object, that's it...

:)

No idea :) i don't use Windows, and i don't recommend you to use it. I recommend using wsl 2 so you can use linu instead.

I highly recommend to install "Git For Windows", it comes with a Linux-terminal emulator called Git-Bash. Having Git installed is always a good idea anyway, and with Git-Bash you can use most basic Linux commands, like "time" for example directly in Windows. That's what I do. I don't like to work from inside a virtual machine.

What version are you using?

Glad it did :)

It depends on your Angular version. Angular 8 and below we need $event. Recent ones doesn't need it anymore, remove it and it will work

I'm glad you liked it :)

That's why i gave you the github repository :) compare your code with mine if you need to :) Best of luck

@@codeacademia i found the problem. Some configurations at tsconfig.json. for some reason it was in strict model. I change mine for yours and it worked