Realtime Advertisement Clicks Aggregator | System Design 

Your videos are really really great, no fluff, straight to the topic and covers a lot of details. Thank you and keep it up!

Thank you so much for this perfect explanation!!

Very organized and neat! Thank you

Thanks for the effort making this! Very informative and a perfect companion to the system design volume 2 book.

I think you deserve a lot more audience! The quality of the contents were really good. Thanks for sharing.

Great explanation and examples !

what I would've done differently: have both warm and cold storages. If your data access pattern is mostly reading data from the last 90 days (pick your number), then store that data in warm storage like Vitess (shared mysql or some distributed relation db). And run a background process periodically that vacuums the stale data from the warm tier and exports it to cold tier like data lakes. This way your optimising both read query latency and storage cost. Best of both worlds.

This wasby farthe best video..thanks for dojng it

Excellent

Thanks for the tutorials! I think you're following the topics of the book System Design Interview 2 but using a way that a lot easier to understand. I'm very much struggled with those topics of the book until I came across your tutorials!

Some notes about this design - Adding more topics is a very vague statement. We have to define the data model to capture each click event and then allow data partitioning based on advertisement_id and some form of timestamp - Not sure why replication lag is stated as an issue here. The read patterns for this design doesn't require reading consistent data. So this should not be an issue - Relational DBs won't do well with aggregation queries. This is a little misguiding. Doing aggregation queries efficiently requires storing the data model in a column major format that unlocks efficient compressions and data loading. - Why provision a stream processing infra to upload data to cold storage . Once a log file reaches X MB, we can place an event in Kafka with a (file_id, offset) pair. There would be a consumer that reads this and uploads the data to s3. This avoids un-necessary dollar cost as well as operational cost of maintaining a stream infrastructure.

awesome

Each day click storage = .1KB * 3 B = .3 TB/ day and not 3 TB /day ? Correct me if i am wrong

We should use count-min sketch for real time click aggregation on the stream processor, it is going to be very fast and you query data on last minute granularity. A map-reduce system can be useful for exact click information. Clicks can be batched, put into HDFS system, reduced into aggregates and saved on DB.

Event data stream platform . It’s more complex system, where data is being processed either in real time streams or batch, ETL, data pipelines etc

Since am working in adtech and would looking to upgrade our approach to modern, fortunately i got a look into your video and it helps me a lot. My question here is how about to use Clikchouse instead of Casandra, will it work well or lead to any issue?

capture the click with application logging, good idea, main crucks 6:30 21:30

You videos are great!Very clearly articulated!Was curious why do we have to use Nosql DB , if we are storing only the aggregated data based on advertiser ID.What are the drawbacks of using any columnar DB like snowflake in thise case?

Thanks for such a clear and detailed explanation. Could you please share a couple of blogs/articles for reference where companies are using this kind of systems?

Thanks for the video, that's brilliant! But I didn't quite understand the problem with inaccurate data in the stream process. Late data can arrive to the stream process, but I suppose it has timestamp from apache log. So can't we just insert that late data to Cassandra?

That was an awesome video, i had a similar approach and got it validated. I was wondering if you could also start a code series on building such systems (as demonstrated in video).

Hello, when we introduce a log file, how do we ensure the aggregates is still "near realtime" ? IMO when you introduce log files in the middle you will have append only logs which we will probably only publish once the log file finished appending and started generating new file. So there is a delay of sometime may be a min or something (depending how big is your rolling window).

Big Fan! hopefully lot more people find these. but just one feedback. you covered what kind of data we store. would it also be worth going little deeper into data model?

Great Video! A minor error: 0.1Kb x 3B = 300 Gb

The tutorial is helpful. Could you give the sample of the format of the records stored in Cassandra. Do we need to store the data in Cassandra with all data in original click?

Can we use MapReduce for stream processing? Will it meet the latency requirement? Or we have to use some other streaming processors such as Flink/Spark?

Correct me if I am wrong, Seems to me more like lambda architecture.. aggregation being fast but inaccurate whereas S3 being slow but accurate

Excellent video from thinking from basic design to scalable design. Few question. 1. what technology actually we can use for log watcher, 2. can you please correct me about stream processors for saving data in s3, we are using the Steam processor. Here stream processors can be any consumer of Kafka events like my simple Java service which pushes data in s3 and for stream processer for aggregation of the data and saving in Cassandra we can flink.

We could also keep States in Kafka Stream application (local or Global State) and use Interactive Query to fetch result of the aggregation. Can you please share how to decide whether to offload the aggregation result to external DB vs when to use interactive Query ? I understand that durability can be one factor but what are others ?

Great video! Did you create these diagrams in obsidian or these are images?

@5:35 0.1KB*3B = 3 TB Hi, how is the computation done? I thought 3B is 9 zero; multiply it by 0.1 will get 8 zero. 1 TB is 1e9 KB. Then I thought it would be 0.3 TB. Did I get something wrong?

For batch job, i think you need separate stream processor, you cant mix real time data and old data. Please share your thought

Correct me if I'm wrong - we are processing the data fetched via Batch Job, so essentially we are processing the data twice to get rid of the inaccuracies ?

How are we using log files?Reading from log files is the best way to source?

Thanks for clearly explaining the end to end design. Just a couple of questions: 1) Could you explain a little bit about how the Apache log files gets the clicks information and how is it realtime. 2) Also, Do you have any link of these notes/Diagram. As the one in description doesn't work.

Is it correct to say that Kafka is Data ingestion platform as data is getting stored in the topics which would be later processed by stream processor?

What do we store in S3? Apache log files?

Given that Kafka is durable, why are we storing the clicks in Log Files? Would this not hamper with system performance? If we need logs, maybe the Kafka consumer can log it to files.

Is it 300GB or 3TB ?

can someone help me with the cassandra database schema design ? like what all tables will be there and what all columns will be there?

What about expenses calculations? How to estimate how much this system will consume money?

Why can’t we process the clicks a bit later than they happen so we capture late data and avoid the batch job? This wouldn’t be 100% real time, but do most systems need super fast real-time processing for ad clicks?

Watched. --

I can't download your pdf Notes