Haven't finished the video yet - but wanted to mention that I've sometimes had GPT behave a little oddly when I specifically use the word 'crawler' in a prompt. It's a bit like it goes into "I shouldn't really be doing this.. mnnghhhhhh!" mode. But telling it that it's a super-duper-wonderful positive thing to help a visually impaired user navigate the web works more reliably. Not sure if it's just random chance - but seemed to work at the time.
Amazing! Always fun to see real time problem solving and how to go about it! Just saw the web crawler used by Jason AI in his latest video...been following both of youll since april... thanks to both !! 🍻🍻
Yeah I'm having some code written for me with selenium uh if you can give me an idea how this is better these are kind of long tutorials but that would be great if you could give a little synopsis
Great video! I am now a fan. I have pull the code and the link clicking failed each time. I suspect node version maybe. What node version are you running?
One of the issue I ran into with a similar project sending website screenshots to gpt-4o had to do with long web pages. They would generate long skinny images, which were unreadable by the AI. As I understand it, screenshots are resized on OpenAI's end into a square (ie: 1024x1024), maintaining the original image's aspect ratio. This results in a lot of the text being unreadable (too small). I've tried splitting these long images into part_1, part_2, etc, but it obviously results in some images getting split in nonsensical areas, which also causes problems. Would love to hear your thoughts on this.
In my video "5 Use Cases for GPT-4 Vision API" I scrape an Amazon search results page by splitting the screenshot into parts, but I also "overlap" the parts so that no product gets cut in half. Depending on the specific website you're scraping something like this might work.
