Very helpful video, although I am struggling with the deployment stage. When running netlify deploy, it just seems to hang until eventually I get a JSON HTTP error with a status code of 500 and the logs on netlify just say cancelling deployment and I'm not sure why. Following your example, I don't have a build script since my app runs the same as yours, however it seems like when attempting to deploy from Github instead of the CLI, netlify runs a build command, which we don't have in this case.
Thanks for the tutorial, I get all the way to the point where I test to return the title but I get this "message": "Protocol error (Page.navigate): Target closed.", I put await on mostly everything that I can see would need to.
hey you should check out this newer video i published recently, it may be helpful to get this working properly: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-kOdIzhPfLuo.html
Hi, Colby thanks for this tutorial, fantastic post. I'm trying to port this workflow to Nextjs. Firstly I tried as is and was getting a bunch of errors I then tried using puppeteer (as opposed to puppeteer-core) this worked locally but failed when uploaded to Netlify and Vercel's servers. Any thoughts, please.
hey James, thanks! i dont have much context here but when deploying to cloud providers you need to be able to include a browser to be spun up virtually, thats where the chrome-aws-lambda comes in: github.com/alixaxel/chrome-aws-lambda - sounds like that might be the issue? you can see my setup and configuration here: github.com/colbyfayock/my-puppeteer-function/blob/main/functions/results.js
@@jamesgrubb not totally sure off the top of my head. Did you happen to see any error logs? If not you could try adding some random console logs or try catches and see where it's railing
@@jamesgrubb I wrote this a while ago that gets deployed to Vercel but as you can see here I used the Vercel function pattern and deployment instead of nextjs. Can't remember the reason behind this but that might be a sign github.com/AutomationPanda/applitools-preview/blob/main/functions/eyes.js
Do you have any guidance on setting this up with CORS? Right now, when i try and pass data in the body of the request or just try and ping the endpoint, I get a CORS error and the function doesn't work. This happens on localhost and live.
Hey really awesome work!! I'm still new with back end stuff, I really wanna know if I can implement this to my chrome extension, I want it to show the data on other website(adding element div using chrome extension stuff) using headless browsing but need it to be interactable like "VehicleStatus("DJAI" ,"3C6MRVJGOME571321") " and will pull up the status on the spefic page I wanted
hey not really sure, havent worked with chrome extensions, but you could totally accept a body in the serverless function and pass that into whatever logic in the headless browser session to find what yo uneed
what about netlify functions pricing? can you give an example how it is gonna be if we scaled scraping puppeteer with netlify how much does the free plan will last
can you elaborate on what you're asking? most providers have a limit for how much time you're able to keep a function running, but generally speaking, as long as you're within that time limit, the function shouldn't just shut off until you respond or hit that timeout
@@colbyfayock Gladly. Here is an example: An application needs to use Puppeteer to sign in, get request headers and use them to make requests to non-public APIs. The catch is, is that the sign in session needs to be kept open to allow for a user (on some random front-end) to make a request to fetch that non-public API data using the session from signing in using Puppeteer.
Had this working for a few months, then came back to my program after a break and now I'm receiving an Error: Navigation failed because browser has disconnected! .. Happens in dev, and made sure my executablePath is pointing to my Chrome app. Any help would be greatly appreciated 🙏🙏🙏 Thanks for the great video!
@@colbyfayock I just checked earlier today and although it doesn't error out in prod it doesn't technically work. It's supposed to post data to my database but doesn't
@@raze_quaza maybe you can try adding console logs to look at the function logs? you can add things like try/catch around the requests / functionality to see if any issues pop up. i also like to add console.log checkpoints to see how far it got before it failed
@@colbyfayock I'm new to Netlify. When I saw async in the function I thought that meant background function. Since from my reading functions are synchronous and background functions are asynchronous. However, I realize now all functions re marked as async.
@@scottfwalter yeah they're synchronous in that they'll run as soon as you make the request and return a response in a synchronous-like way, but that doesn't refer to whats happening inside of the function
@@jamalmriley it might be the node version. last time i tried this with the same package versions, the node version gave me issues, particularly trying to deploy with that version / make other packages work with it
Great video! However while everything worked locally (Windows), they didn't work when the app was deployed. I'm getting this error, probably related to Chrome. Error - Failed to launch the browser process! /tmp/chromium: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory TROUBLESHOOTING: Here are the versions of the dependencies in case this is the problem: "dependencies": { "chrome-aws-lambda": "^10.1.0", "dotenv": "^16.4.5", "puppeteer-core": "^10.2.0" }
hey, it's been a little while since i played with this, but the last time i tried checking this out part of the issue is some of these dependencies require older versions due to the bundle size and those older versions require old versions of node which are deprecated for at least Netlify and Vercel from my understanding
@@blackjack4482 The problem is an issue with the f***in Chromium versions. This package seems to solve the issue. Try these dependencies: "dependencies": { "@sparticuz/chromium": "^110.0.1", "dotenv": "^16.4.5", "puppeteer-core": "19.6" } And I think you might need to adjust a method a bit to match their syntax. I think something like this will work: const browser = await puppeteer.launch({ args: chromium.args, executablePath: await chromium.executablePath(), headless: true, });
