Shopify's Architecture to Handle 80K RPS Celebrity Sales • Simon Eskildsen • GOTO 2017

Подписаться 990 тыс.

Просмотров 31 тыс.

50% 1

This presentation was recorded at GOTO Copenhagen 2017
gotocph.com
Simon Eskildsen - Production Engineering Lead at Shopify
ABSTRACT
What do you do when some of the most ubiquitous celebrity personalities launch products on your platform, driving tens of thousands of requests per second? You pull up your sleeves and architect for it. Throughout the past decade, Shopify's infrastructure has evolved to serve some of the largest online sales on the planet.
In this talk, we dive into our multi-tenant architecture that allows us to failover between regions with zero downtime, move shops between shards, minimize [...]
Download slides and read the full abstract here:
gotocph.com/2017/sessions/161
/ gotocon
/ goto-
/ gotoconferences
#ShopifyArchitecture #Shopify #SimonEskildsen
Looking for a unique learning experience?
Attend the next GOTO conference near you! Get your ticket at gotopia.tech
Sign up for updates and specials at gotopia.tech/newsletter
SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily.
ru-vid.com...

Наука

Опубликовано:

17 окт 2017

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 17

@becauseoffreedom 6 лет назад

Hello SImon, just a few questions. 1. How do you guys update your code base, database structure changes and config for all the pods without downtime and issue? 2. How do you backup your pods in the dc level? 3. If your load-balancer gose down, does that mean entire shopify shops worldwide go down? 4. Is pods = docker container or kvm instance? 5. Do you sync all pod's mysql into kafka in the end?

@TBasianeyes 5 лет назад

A bit late and I don't work at Shopify but should still be accurate: 1. The code runs in Docker containers, so during build they package the application with all its dependencies into a Docker container and push that to a Docker registry. These containers run on Kubernetes (a container orchestrator) which will also take care of deploying them without downtime. The deployment does a rolling update of all containers running the specific type of container, this means that they are updated one by one. Shopify has open-sourced a part of their deployment tooling which is kubernetes-deploy, it is a tool to execute and watch Kubernetes deployments, you can find the project on Github. Database changes are done in a way to be compatible with the old and new codebase, sometimes requiring multiple deployments to get to the final desired state. They run as part of the deployment before deploying the new application code. This order of execution is part of kubernetes-deploy. 2. Data doesn't live in pods, pods only run code. Data lives in databases and other data stores so there you apply good old backup mechanisms like snapshots. 3. No, as was explained in the presentation the system runs in multiple regions and each region has redunant load balancers. If a region goes down it doesn't get announced and no traffic reaches it. Shopify also runs their core in 2 completely region isolated data centers and sync all data. 4. Pods are Docker (or containerd) containers running on Kubernetes. 5. No.

@riadassir5896 Год назад

Excellent presentation. Very useful. Thank you

@tharun8164 10 месяцев назад

Nice talk. A question. During mysql failover(DR ?), you mentioned about handling requests in a queue and then responding the client using throttle. Does this happen in synchronously? If yes, then how?

@cybernetic100 6 лет назад

Great talk Simon! Couple of questions: 1. You use the bin log to stream writes over to the new db when moving a shop from one pod to another but for the data that doesn't change all that much like, may be, products do you still copy them over using the "naive" approach? If so, then is there any locking involved with that? 2. You mentioned that shops share the schema, does that mean all the shops use a single physical database with logical partitioning on shop id? or each shop has its own database but they just happen to all have the same schema for predictability reasons? If the former, how do you prevent the single database becoming a bottleneck? Thanks

@hormozkheradmand9186 6 лет назад

(1) Simplified slightly, we use a hybrid approach of copying data using SELECT+INSERT statements and using the binlog to apply live changes. Once we have "caught up" with the binlog, we set the shop to read-only and perform a cutover. (2) The former is our approach. We horizontally scale shops on multiple databases once a single database has too many shops

@cybernetic100 6 лет назад

Ok. so if I understand correctly, if the database has say, ShopA, ShopB and ShopC and that's 3 shops too many with ShopC being rather large, you would split out ShopC into a separate database (with identical schema) and then route traffic accordingly?