Demo: Rapid prototyping with Gemma and Llama.cpp 

I've used gemma for a benchmark in a research project I'm working on, where I compared human results against AI, gemma was the closest after bloom 176B, followed by models like mistral instruct 7Band llama 34B, even the 2b version did pretty well, great work team 👏🏻

Very cool, thank you! I like this format with demos. We are developers!

Thanks for the demo!

Thank you for the discussion.

Really cool and simple to understand

This is a fantastic format for a talk, insightful and engaging for a viewer!!!

Wow this is so cool 😎, developers changing the world

Like it very good sharing.

Good demo, nice tooling suggestions out of this

Cool demo, I'll try it

I was looking for this! Thanks! 😎🤖

Very nice 👌 cool Demo

Minimum System Requirements for running a model locally?

This is so cool.

Very nice!!!!

This is so cool :)

how do you connect the LM Studio to llama.cpp? you used a preset which was probably custom.

Wrapped in the llamafile runtime it is an even better single file .. oh yes!

cool demo

At 9:03, Gemma shows some solid satire

good, thank you

Cool. Will definitely use this soon

CODE would be awesome, as well as knowing the SPEC of the notebook as it’s easy to buy a sub1k or 5K+ at apple store depending on so many factors…

Great choice of demo app!!

Very interesting demo! What kind of extra work would be required to run this without LM Studio? Does Llama.cpp contain the necessarry functions to load models as servers you can interrogate?

Is this meant to be super accessible? If I have an APU, on a laptop with no GPU or NPU(?), that means I can expect it to run fairly well? Also considerations for a lighter yet usable model?

How about windows environment? Is LMstudio/Ollama a better choice?

If anyone faces an error on Mac about "network error: self signed certificate", close the app and use terminal, run "NODE_TLS_REJECT_UNAUTHORIZED=0 open -a "LM Studio" " This reopens the app and the error should go away. I have no idea where to put this info sooooo...

Very nice, but what is your computer spec? Memory and chip?

Has anyone else tried doing this? I tried following this exactly with LM Studio using the exact model and prompt but I am consistently getting atrocious outputs; the gemma model is just outputting gibberish or incorrectly formatted JSON. I wish there were more details on the presets used.

Very cool demo 👍. Any chance we can get the source code 😀

where can we learn to set this up ?

Anyone notice ram usage of 16.68 GB? What was that?

wow

Just my two cents: 1. No explanation on how to connect LM Studio to the Llama.cpp, 2. newest hardware required - at least it doesn't work on my M1 with eight performance cores and 32 GB Ram

did gemma should use GPU ? so I wanna try to learn this model, but I didn't want to use GPU

Awesome.

What’s the required hardware specs?

damnn its using 15 gb of ram i have an 8gb m1 i dont think it will work for me..

That's a lot of RAM to run this for me.

Pls provide source code also

game using lm studio, interesting.

It's cool now, but what if each application starts deploying local models. It'll turn our phones into what data centers were meant for thereby reducing costs for large corporations. Trading a few megabytes for faster and more expensive chips.

never trust google products.

Just focus on your existing products Google. Which are a mess 😂

Google seems to be trying out too many new things. to me thats a turn off and red flag. everyone is different 🙂