One remark here. Bull is not exactly the same thing as BullMQ. BullMQ is a newer version of Bull written in TS. NestJS provides separate packages for Bull and BullMQ.
I believe the consumers should be an app separate from the API, no? The way your are doing in the video, if you have 5 replicas and send 5 requests, a 6th request will hang because the transcoding processes are blocking all 5 API nodes.
Well, actually, you can use the same app but offload the transcoding to a worker thread which is designed for CPU-intensive tasks, or you can spawn a process if you are using ffmpeg, for example. In this way you can put in the bull job the code that spawns the new thread/process and you will able to scale for an infinite number of concurrent transcodings, because bull will send the next job to process only after it finishes the previous one (unless you overwrite the default concurrency).
Hi Michael, very nice content!! Just want to point that you need to create a LB (eg nginx) to randomly pick a pod inside the k8s and make sure that you are indeed using a distributed system.
This is great! I see however usage of Bull and Redis is mostly recommended for intensive tasks. Would implementing this queue architecture on, say a REST service, beneficial? I'm thinking maybe the queue system can help on handling errors and probably restarting failed tasks that run daily on the application, such as a data validation task or a database mutation.
So instead of using Docker and K8s orchestration what if we're using a LB with multiple ec2 instances (like in AWS) would the entire setup still pick only 1 of the consumer to process the message and not multiple consumers (from other servers) fighting to process that message? I mean is this the nature of distributed queuing system that allows this to happen or something else?
Hi Michael, great video again. Very well put together, straight to the point with a real example. If you don't mind me asking, what are the reasons someone would choose to use bullmq over say rabbitmq? Is there a particular reason you have started looking into it?
So I think they're different solutions to this same problem of achieving scale by distributed async processing. They obviously have their own differences between each other and it's a matter of finding what works best for you. BullMQ is "simpler" in my opinion, no need to ack messages, etc.
How you think this great solution compare to using an orchestrator like Netflix conductor or Netflix maestro? Thanks for sharing your knowledge with us.
For a newbie in Nest I didnt get quite a number but somehow this ia looking so cool and helpful.| Cant wait to flow in Nest and deployments with K8s like you did on this project. Thanks mate :)
Great content again. Quick question though, with your first example "transcode an audio file", why choose to go for a job with BullMQ (the queueing system with Redis) instead of an event with EventEmitter (appart from showcasing it of course)? Both would achieve the same result right ? Not blocking the thread and decoupling the producer/emitter from the consumer/listener ? My question is thus : are those 2 patterns just different ways to implement a distributed system ? Why go for one or the other then ? What are the main differences ?
There's is a fundamental difference between a QUEUE and a PUB/SUB or EVENTS. A queue (just like the name implies) strictly follows a FIFO sequence, meaning that, regardless of the number of replicas or instances there are, only one of this instance can dequeue and process the data in sequence (based on how they got enqueued). However, in the case of events (or pubsub), there's a producer of the event and potentially multiple subscribers (one to many). Hence, when an event is published, multiple instances could subscribe and react to that event. In essence, it depends on your architecture and what you're trying to achieve but that is basically the difference.
