Thanks john for giving us this valuable information, But one thing that i want to understand How you design the multiple functions in web scraping project and how you make the relationship between these functions . what kind of strategy or approach that you would use. or what kind of things that i should learn to make any kind of project ? (Process of designing the functions).
i think the best way is to just let it naturally happen, when you find you could abstract some code to a function try it and see how it looks and feels. Generally I have at least 3 functions, one to get the data, one to parse out what I want and one to save it somewhere. start with those and work out from there
But isn't this also achieved by using threadpoolexecutor with the max_workers option to control the exact number of requests sent to the server at a given time?
This is used for asyncio so you'll probably have a better control of concurrency than threadpoolexecutor (thread vs co-routine). It's likely that the requests being sent will be controlled if something fails, that won't happen in threads.
Hi @JohnWatsonRooney, Could you make video on implementing rate limit for web sockets in chatbot using python, FastAPI. If anyone knows please share it your suggestions. Thanks
Exactly. I had hard time with making fast and stable async requests and the best solution I came up with was Semaphore. Limiter works as well but I find it slower.
john , I have watched almost all of your videos, I found that all the content has a separate demo, all the code is written together, hope there is a framework design or a complete project of this kind of tutorial.thanks
Hi there, nice video! With a very similr problem (and to avoid server abuse) I encountered the same issue, but ended up using asyncio's Semafore, with the expected outcome as yours! Also, have you considered adding uvloop as your DefaultAsyncPolicy? That little thingy gace me very good results lately. Cya in another video 🦾
