Web Scraping AI AGENT, that absolutely works 😍 

Someone noticed it :D

I am not having success with it. It only gives me urls, titles, related posts. No content that I ask for.

Haha

Glad it was helpful!

What is an octopus? Which crawls API's or do datamining

No, it's my bad. After the video I reviewed the web page. In fact, I added the screenshot in the video. It was inside the carousel

Thank you so much 😀

🕯️🕷️🕯️

Honestly, I was actually scared before uploading this, but let's see!

@1littlecoder Hopefully all is well.

Thank you 1littlecoder@gmail.com is my email

Do you want me to show how to run the code from GitHub? Will it be helpful

Yeah! At least in a way that's easier to understand. I don't know anything about code, so I need things to be clear and simple.

Thanks!:)

I'm sorry I had done it a few times before so didn't repeat ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-C0GmAmyhVxM.html

@@1littlecoder cool. Thank you 👍🏻

it is the same error or not here: I'm very beginer level if someone know how to fix mine please leave the comment --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[25], line 17 3 graph_config = { 4 "llm": { 5 "model": "ollama/mistral", (...) 13 } 14 } 16 # Instantiate the SmartScraperGraph class ---> 17 smart_scraper_graph = SmartScraperGraph( 18 prompt="List me all the articles", 19 source="news.ycombinator.com", 20 config=graph_config 21 ) 23 # Run the smart scraper graph 24 result = smart_scraper_graph.run() File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\scrapegraphai\graphs\smart_scraper_graph.py:47, in SmartScraperGraph.__init__(self, prompt, source, config) 46 def __init__(self, prompt: str, source: str, config: dict): ---> 47 super().__init__(prompt, config, source) 49 self.input_key = "url" if source.startswith("http") else "local_dir" File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\scrapegraphai\graphs\abstract_graph.py:49, in AbstractGraph.__init__(self, prompt, config, source) 47 self.config = config ... --> 227 params = self.llm_model._lc_kwargs 228 # remove streaming and temperature 229 params.pop("streaming", None) AttributeError: 'Ollama' object has no attribute '_lc_kwargs' Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...

How did I not even think about it?😭😭😭

@@1littlecoder :)

What was ur requirements?

@@webhosting7062 I write a daily report, based on the new posts in various FB groups .. but FB doesn't put posts in the correct order (also, pinned posts up the top will be old posts) .. so i need to check the date, but, FB obfuscates the date like a MF .. i wasn't able to figure it out with selenium. so, requirements are .. 'get the latest (less than ~24hr old posts) from a FB group.

Glad you enjoyed it

Plus RAG, yes!

ollama is just a frmework to run LLMs locally, so it downloads the model insted of using an API and connecting to server

If you just want scraping, don't bother with this. However, if you want scraping + RAG, with LLM integration, then use this. But it's not without it's issues

I guess mine is 3.9ish

😭 will try to fix it!

You can run on colab. But you'd need openai keys

I think in those cases you can probably use a conventional libraries I guess but that's a good question there are different classes within this library that might let it do

@@1littlecoder from scrapegraphai.graphs import BaseGraph from scrapegraphai.nodes import FetchNode, ParseNode,generate_answer_node graph = BaseGraph( nodes={ fetch_node, parse_node, }, edges={ (fetch_node, parse_node), (parse_node, generate_answer_node), }, entry_point=fetch_node ) .. i dont have time to try it now cause im at work :))

I've not tried it! it'd be a good opportunity to try that, especially given it uses Playwright!

@@1littlecoder ill tell ya, i tried and it fails miserably :)) , if you have better luck let us know man

@@adriangpuiu ah that's bad. Which website was it ?

@@1littlecoder the user replyes are incapsulated in a JS response from what i noticed, maybe they have an api or soething , i was just unable to figure it out . YET ...

@@1littlecoder its the appian discussion forum

I have not tried it . Someone else in the comments said it might not very good.