It is a shame that the lazy API is so entangled into the API. Might be nice to write generic code which then has the option to switch on the lazy API with one single change. I don't like the idea of having to rewrite the whole codebase to switch between lazy and eager. I question if that was a good design decision from polars. - Side note please always upload videos in 1080p
📝 Summary of Key Points: 📌 Evaluators for AI Models: The workshop focused on the importance of evaluators for AI models, especially large language models (LLMs), to ensure they are performing correctly. 🧐 Types of Evaluations: Different types of evaluations were discussed, including human review, user feedback, implicit user feedback, comparing against a reference, and ungrounded evaluations. 🚀 Building Evaluators: The workshop delved into building evaluators using LLMs, breaking down tasks into simpler steps, and utilizing techniques like faithfulness evaluation and context relevancy scoring. 💡 Additional Insights and Observations: 💬 Evaluation Techniques: Using LLMs for evaluations simplifies complex tasks, and keeping evaluations simple reduces chances of errors. 📊 Cost and Latency: Evaluations are recommended for analysis and development rather than real-time applications due to latency concerns and API costs. 🌐 Custom Evaluators: Consider using functions as evaluators or text embedding-based approaches for cost-effective evaluation strategies. 📣 Concluding Remarks: The workshop emphasized the significance of evaluators for AI models, highlighted various evaluation techniques using LLMs, and suggested cost-effective evaluation strategies for robust environments. It provided insights into setting up evaluators in CI/CD pipelines and the importance of simplicity in evaluation tasks. Generated using TalkBud
There is a ton of tabular data that businesses have. Creating knowledge representations out of this with knowledge discovery, using reasoning and inference on this and then managing the life cycle of these representations is what we need. This is a use case specific workflow, requires domain and AI expertise. Natural language representations are great, Chat GPT is great, but... we do have a ton of tabular data too. Why go through and rediscover knowledge representations for these problems since chat GPT came along.
That's a great point Rajiv! Just because LLMs have unlocked many unstructured data for us doesn't mean we should abandon our existing structured datasets.
Amazing stuff Shaw. I watched your entire video series. My self-assessment of skills: High expertise in a service company working mostly with contracts and third parties, and medium technical knowledge (writing my Master's thesis in LLM, concept knowledge, Langchain, some Python and so on). What is the best way on hugging-face to fine-tune a model based on, say, only 10 real estate contracts written in Word to create a text generator based on a specific fact, like lets say: "The tenant is not obliged to rebuild, but must {...}, where brackets represent the text to be generated. Any input is highly appreciated :) Thank you so much! Followed on medium as well...
Thanks, glad you enjoyed the series 😁 At first glance, 10 examples is not enough for effective fine-tuning, but it’s definitely worth a shot. I’d recommend setting up time to chat during “office hours” so we can dig into this a bit more. calendly.com/shawhintalebi/office-hours -Shaw
Good question. Here’s a response from Ben. “I’m not entirely sure tbh. i'm pretty sure pyspark is more scalable (e.g. > 1 TB data), but polars is better for data processing on your local machine (e.g. < 1 TB). i don't think Polars has so much stuff yet like pyspark does for distributed computing, whereas that is pretty much what pyspark was built for afaik.”
Talk talk TALK talk talk tALK talK *TALK*! NO code, no realistic use cases and examples - none of that. I can't get those 45 minutes back, gentle reader, but now you don't have to lose them.
Thanks for the feedback. This talk is definitely more high-level and conceptual. Given the popularity of the topic, we'll need to bring Jeremy back for another more technical and hands-on workshop.
I use PySpark and Pandas every day at work, and am interested in learning more about Polars. I've messed with Polars a little, and it kind of seems like a cross between the two. Looks like Polars has window functions, which is something Pandas is missing compared to PySpark. One issue I see in general is there's a whole lot of really bad Pandas code out there, often written by people coming from Matlab who don't bother to really learn Pandas before deploying code. So be really careful of Pandas code you get out of SO or ChatGPT. And your Pandas example at 17:22 could be vastly improved by dot chaining, which is hard to show in YT comments but I'll give it a shot: decade_counts = ( pd.DataFrame(...) .assign(decade = lambda x:x['Age']/10*10) .groupby('decade') .agg(n_observations = ('decade',len)) .sort_values('n_observations', ascending=False) .iloc[:10] .plot(kind='bar') ) I added the lines starting at sort_values to show how to plot a top 10 bar chart of the data. Key here is to dotchain and avoid assignments.
@@TheDataEntrepreneurs yeah, I just don't see it. With every other revolution there was a place for humans. Now we're no different than horses when the car was invented. This is a GOOD thing. All that matters is that world assets continue to rise in value. In the short term I think entrepreneurship will be at its best and in the near short term we'll just all be free to do/create what we want. I don't see a world in 5-10 years where anyone works for anyone.
Although I see the benefits of Polars. I haven't had enough obstacle with Pandas for my workflows. I don't deal datasets that exceed memory and I think currently I can extend my memory limit using Dask ... but looking forward to the development of Polars and will likely adopt once it has more support!
Chatgpt knows pandas much better. For exploratory work probably not an issue. But if shopping something to prod and want to keep it very fast and minimise system resource then polars seems a better choice.
To me the appeal is the coherence of the API and the superior execution model. But the ecosystem disadvantages associated with using a much less popular library are substantial.
@@samuelswatson but polars has a to_pandas() method, so the disadvantages is easily overcome, so its more like if you your doing simple things, then its unneccesary to bring in polars. We replaced a lot advanced elt(not etl) with polars. 16 000 lines of sql code done with the main transfroms done in polars instead, for this task it's excellent and translated really well, and a lot of stuff is easier to to in polars than in sql for example. Doing the same in pandas is a nightmare. Translating advanced sql code to pandas is a hard job.
@@signoc1964 That seems to me to be the best use case for Polars (replacing complex SQL in transformation pipelines, especially because of its composability), so it's cool to hear another testimonial for its success in that context.
It is a very informative video, thanks a lot, Shaw. Regarding the LLM business opportunities, I think the biggest opportunities in the field for the next decade are to help companies get LLM-ready data sets. The computing costs will eventually decrease with more companies joining the GPU race. Companies will realize their data is valuable and want their own small model adapted to their own use cases (vertical + sector). However, only a small proportion of companies have a clear data strategy and have their data ready for LLMs. So the opportunity is huge for companies like Databrick as well as for vendors who can just jump in and help companies sort out their data.
That's a great point. As organizations become more interested in developing custom LLMs, there will be a large demand for data curation and preparation. Thanks for sharing your insight!
Pandas is a massive mess. It's very easy to write very poor code in Pandas but to write it well is not that intuitive, Matt Harrison has written some great stuff, but it's not that easy to learn. Polars is way better and improving all the time. It's much easier to write and way quicker. The lazy API is a thing of beauty.
I agree, Pandas' learning curve was steeper for me than R. But I've kind of settled on a consistent workflow(.loc[:,['col']] instead of ['col'], and dotchaining) that minimizes the surprises.
One problem with polars though is that "pandas" developers then to write "polars" code like they write pandas code, and to some extent it is possible which gives people a bad example, since a couple of those. Polars becomes like pandas then executing in serial instead of parallell.
Hi. You say to not do the transformations in feature engineering pipeline but include steps like "standardizing" and "vectorize". let's say I have some text data and I'm using a bow vectorizer. How can I include that in this FE pipeline? This is ideally done/fitted on train data if I'm not wrong. How can please explain a little more on that?
Paul dives into more detail on FE pipelines here: towardsdatascience.com/a-guide-to-building-effective-training-pipelines-for-maximum-results-6fdaef594cee
