@Scale is a series of technical conferences for engineers who build or maintain systems that are designed for scale.
Building applications and services that scale to millions or even billions of people presents a complex set of engineering challenges, many of them unprecedented. The @Scale community is focused on bringing people together to openly discuss these challenges and collaborate on the development of new solutions across a series of topics.
I would like to speak with somebody personally about this workflow processes on Facebook I'm receiving messages about my business page saying they may shut me down because of logo and infringement but that logo is my logo which I've paid for. Can you help me with this situation? @Scale
"it's very common for both old and new code paths to exist in the codebase simultaneously controlled by the use of conditional flags" Dear god what a fricken nightmare.
Thank you for the information. It would be really interesting if there was a way to use the rejected heat to purify water. An example is like naval vessels use vacuum distillation to purify sea water into fresh water.
Amazing work, thanks for the demo at 12:55. MLow sounds amazing much better than Opus 6kbps, although I was hoping for more comparisons with other low bandwidth codecs.
This is an excellent video. Maybe change the chapter breakdown to "Video Backend Infrastructure" instead of "Jithin Thomas" at 08:20? Overall, awesome. Cheers!
🎯 Key points for quick navigation: 00:19 *Meta transitioned AI training from horizontal to vertical scaling, requiring a dedicated RDMA network over converged Ethernet.* 01:00 *RDMA fabrics at Meta support tens of thousands of GPUs, handling diverse AI training use cases.* 03:11 *AI training involves complex, recursive processes that scale vertically with HPC-style parallel processing.* 05:32 *RDMA with RoCE V2 enables high-bandwidth, low-latency GPU communication crucial for AI training.* 08:30 *Meta's network design for AI training includes balanced topologies and traffic patterns accommodating hierarchical and full mesh models.* 12:44 *Load balancing challenges in RDMA deployments at Meta involve adapting to uneven distribution of server destinations across IP prefixes.* 16:52 *Issues with slow receivers impacting network performance at Meta are often related to GPU memory allocation pressures, causing PCI and network bottlenecks.* Made with HARPA AI
Gemini 1.5 Pro: This video is about big data processing and new approaches to data analytics. The speaker, Francis Perry, argues that traditional big data processing methods are no longer efficient given the way data is used today. He proposes a new approach that leverages in-browser databases and cloud storage for interactive data visualization. The talk starts with a historical overview of big data processing. Around year 2000, data centers transitioned from expensive mainframes to cheap commodity hardware. This led to the development of MapReduce, a scalable data processing framework. Hadoop, an open-source implementation of MapReduce, followed soon after. These technologies enabled processing massive datasets but required significant expertise to manage and use. More recently, data analysts are increasingly involved in data processing. However, most data processing only utilizes a small fraction of the entire dataset. Traditional big data processing incurs significant costs to store and manage this vast amount of unused data. The speaker introduces DuckDB, an open-source data analytics engine designed for single-host processing. DuckDB is fast and portable because it compiles to WebAssembly and can run directly in web browsers. MotherDuck, a cloud-based service, extends DuckDB by providing a storage layer and collaboration features. Together, DuckDB and MotherDuck enable a new approach to data analytics. Data analysts can work with data locally on their laptops using web browsers. This approach offers millisecond query latency and eliminates the need for complex distributed systems. MotherDuck seamlessly integrates with DuckDB, providing cloud storage and data sharing capabilities. The video concludes with a live demo showcasing how DuckDB and MotherDuck can be used for interactive data visualization. The speaker uses a Seattle weather dataset to explore temperature and precipitation trends throughout the year. By filtering and manipulating the data, the speaker is able to gain insights into the weather patterns.
Meta, Google, Microsoft and Apple love to talk about climate change but how much energy and c02 emissions do the data farms produce? Not only do these companies destroy Africa for the rare earth minerals with slave labor but then turn around and try to preach about the environment and equality, WHAT A JOKE
In a real case, gigantic company, wouldn't it be a nightmare to compile all the code into a single binary all the time? In my current company, my service takes about 5 minutes to compile and it's not even in the top 10 of biggest services.
You don't build / release the entire repo in one go, but only a tiny part of it compiled down to only a few files usually. So the size of the repo is generally irrelevant. Bigger services are composed of microservice nodes which are owned by different teams and released separately.
