@@codeacademia sorry, actually that was for 10000 objects per batch, for 1000 is around 4s. I wonder if we could do better. Which side represents a bottleneck - nodejs or mongodb inside the container. Also, I am running this on macbook pro with 32GB of memory and around 2Gs of memory for the container. I ran the mongostat inside the container to get some stats - I got around 31,073 inserts/sec.
31k per second is actually pretty descent for 1 cpu core. I can say there is room for improvement if you still haven't reached 100% of your CPU. You can split your data into multiple pieces and run multiple nodejs instances, each one reading from the data and inserting in MongoDb... At some point, you will need to scale your db and make it optimized for heavy write tasks as well!
i have an error on this line this.bigChart = this.dashboardService.bigChart(); : Type '{ name: string; data: number[]; }[]' is not assignable to type 'never[]'. Type '{ name: string; data: number[]; }' is not assignable to type 'never'.ts(2322)
Good catch, default highwatermark in nodejs streams is indeed 16Kb. You can double-check the documentation for that. The number of pipes influences the highwatermark as well as other factors, I'm not aware of it all, but i know from a fact that the size can change, and i let nodejs take care of it and adjust the speed by itself... There is no good/clear documentation about streams in general, so feel free to experiment things :) and make sure you share it with us :)
nice tut question here, what if the master and slave is on a different server, if the master goes down, how does the slave become master gets its data from>?
The master and slave are always in sync. The same data is shared between the slaves and the master. If the Master goes down, Sentienl switches the role of any slave he finds to become Master. If the Master that went down becomes up again, Sentienel will switch its role to Slave.
@@codeacademia cheers for the answer, one issue here is, there is always a permission error, when sentinel trys to rewrite the config, not sure i you get that too, with the latest redis, i used your config to do the test but yeah in the sentinel container there is permission issues
Hii, I'm very thankful for this easy to understand to the point video. But I'm referring to this video in now 2023 and did exact same code but somehow it is just showing blank webpage on Chrome. Any idea or solution to make it show up on screen. It will be very helpful.
@@codeacademia Hey.. Thanks for such quick response. There is actually no error at powershell, also I checked browser console there is none while landing on port but as soon I try to hit some endpoints manually (posts, login) in URL then error appears in console.
@@codeacademia Hi..here is the list of packages and versions I'm using - - > Angular - 15.2.8 Angular CLI - 15.2.6 Node - 18.14.2 NPM - 9.5.1 OS - win32 x64 Angular flex-layout - 15.0.0-beta.42 (deprecated) [ I'm not quite sure if the error is related to this package being deprecated ] The error I'm getting on browser console is - - > NG0203: inject() must be called from an injection context.
You should change the format by yourself. Use Something like Object.values(myObject).join(';') to create a csv, instead of objects use strings ... I can send a code that does that, but you need to learn a bit, experiment things :)
Hey, async pipeline works well on small amount of data (like 20 rows), but in this example if i have 400 lines it stops working and just hangs in processes (jest tests stat to crash due to timeout when i started migrating from pipe to pipeline). Any idea why this could happen? async function transformData() { const sourceFileStream = fs.createReadStream('fileName.json'); const resultStream = new PassThrough(); await pipeline(sourceFileStream, resultStream); return resultStream; }
Thanks for reaching out. Make sure that your last transform doesn't send any data. I've explained that in this tutorial, the last transform should behave like a write stream, its consums the data but doesn't send any.
@@codeacademia So my use case is reading remote file, transforming it and return it as readable stream, PassThrough or Transform can do that and does fine with small file. If i do not send any data returned stream is empty Here is example async function transformData() { const sourceFileStream = fs.createReadStream('fileName.json'); const resultStream = new PassThrough({ objectMode: true, transform(value, enc, cb) { cb(null); }, }); await pipeline(sourceFileStream, resultStream); return resultStream; } The problem is that original example works fine with small file but timesout on bigger one :S Originally i was using sourceFileStream.pipe(resultStream) And everything worked well
That has nothing to do the size of the file. Streams work great with huge files. For your use case use Transform not a PassThrough. And make sure to follow the requirements Await pipeline( yourReadStream, yourTransform, lastTransform) Make sure that LastTransform doesn't send any data in your callback. as explained in the videos :) Best of luck.
@@codeacademia If anyone comes in with this question answer is highWaterMark, because node streams has protection and is build for piping information, it makes sense to have some limit of read data which is highWaterMark. In my case i needed to load all buffer in to the memory, which means i need to have high highWaterMark, which will impact application memory used but lets me pass streams around. Not suggesting to do this to anyone as for my case its bad architecture and legacy code :(
How do you check the progress (status) of a pipeline? With pipe you can use something like fs.statSync and then do str.on('progress'.... and console.log it But with pipeline this doesn't work.
What happens when highWaterMark for the CSV stream is smaller than a single object, would our transform still work? In this case, it seems that each chunk exact matches a single object - is this behaviour implemented in the CsvToJson plugin?
CsvToJson takes care of it, yes. And yes, the transform will continue to work without problems :) I suggest leaving hightWaterMark in its default value.
@@codeacademia how could it continue to work if the chunk you receive represents half of an object or more than one object.. because your transform function expects a single object as input and a single object as output. No?
Under the hood, csvToJson will make sure to send you an object. If it receives less or more, it will make sure to send you a valid object down the pipeline...
watch all your videos, thanks. i'm using ffmpeg with spawn which is piped with one input readstream to stdin of spawn and three output writestream to stdio[3], stdio[4], stdio[5] of spawn. but i don't know how to pipe rs -> spawn -> 3 ws. how can i do it?
I highly recommend to install "Git For Windows", it comes with a Linux-terminal emulator called Git-Bash. Having Git installed is always a good idea anyway, and with Git-Bash you can use most basic Linux commands, like "time" for example directly in Windows. That's what I do. I don't like to work from inside a virtual machine.
Hi, thanks for this tutorial, but i have a problem at minute 35:33 because the property chartOptions: {}; on area.component.ts have the error: "Property 'chartOptions' has no initializer and is not definitely assigned in the constructor." can help to me? i used angular 14
Hello! Can someone help me out? after the first chart, the area one, i'm getting this error: src/app/shared/widgets/area/area.component.ts:11:3 - error TS2564: Property 'chartOptions' has no initializer and is not definitely assigned in the constructor. what have i missed?
@@codeacademia i found the problem. Some configurations at tsconfig.json. for some reason it was in strict model. I change mine for yours and it worked