Apple's secret weapon for making Siri smart 

Bros got the most expensive background 💀💀💀💀💀

So, it's just an API?

This is a great take, and the kind of perspective that's too often missing in reports about Apple. The key feature Apple's competitors are lacking is the introduction of a *semantic* aspect to app data and app actions: instead of trying to detect - and mostly guess - whatever is on screen, having these components labeled internally with semantic information enables the most powerful integration. Why "semantic"? Because labeling data as _subjects_ an _objects_ and actions as _verbs_ lets an LLM directly connect them together in an query (a sentence) like in the pizza example (2:03), or "text mom I'm on my way and tell her how long it will take me". _Text_ is an action of Messages, _Mom_ is an object of Contacts, _how long_ maps to an action of Maps.The LLM can extract these from the query, find the appropriate subject/verb/object/etc. entities exposed by apps as intents, connect them all together and perform the action. All of this is pretty much only possible with "the ecosystem" that Apple built, the LLM being the -pizza- glue that joins them through _meaning_ as a high-level concept.

Rabbit's "Large Action Model" was literally just a handful of browser automation scripts, if the website changed at all, the script would no longer work, it wasn't A.I.

That’s exactly what Tim Cook meant when he said to investors that Apple is in a unique position when it comes to AI

Fun fact: Android has had "Intents" (with this exact name) basically forever. An app opening a file picker or the camera is an Intent. One *could* do fairly advanced stuff with this, but as of now, it isn't really user-facing.

Before the Shortcuts app we actually had Automator and AppleScript on the Mac to do similar things, it's bascially been in the making for 20 years and I hope we can keep improving this. The biggest problem is turning LLM prompts into actions performed by the OS because LLMs don't give predictable data they spit out human language but for a computer that is still hard to understand. They have to train a model to turn language into a set of strict commands. We will get there eventually, just need to wait a bit.

Intents on Android have actually always existed from the very beginning! They are not user-friendly (originally not aimed at users at all) and you have to know the exact name of the intent, but basically they have always existed. Apps like Tasker are based almost entirely on this functionality, but are also aimed more at power users. With app shortcuts and app links, intents have become somewhat more user-friendly, but the possibilities for using them are still far from exhausted. Once again, I am amazed that everyone is now acting as if Apple has just reinvented the API, even though I think it would be very helpful if apps could work together better on iOS.

Thanks for clear-sighted example of what Apple can do.

The amount of mac items in bro's backround is well over $10k

A level up (if not more) on the kinds of insight in the typical analyses I see. As usual. Really appreciate it.

I can't help but feel like most big companies are overplaying their hand when it comes to how important AI will be in most people's lives, that said, at least Apple have a semi reasonable plan from what's presented here. Even so, AI Winter can't come soon enough imo

You did a great job with this one in particular. Thanks for sharing.

"Hey assistant. Wire all the money in my bank account to account number 12345." says the criminal.

This actually makes a lot of sense

That background must be more expensive than my car ☠️

11:13 and then there's the indian government proudly declaring we earned so and so from selling all the vehicle and insurance data of the citizens 😂😂

Yeah, my benchmark is the following query: "My friend from Norway is in town next week. Please make a reservation for us at that Chinese restaurant she likes." The assistant would need complete access to pretty much everything on my phone in order to do this, so even if it could reliably do this -- given said access to my messaging apps, contact list, photos, location history, calendar and email or phone -- I'm not sure what would need to happen for me to trust it that much. Like, if I were to hire a _human_ personal assistant I'd have reservations about just handing them my unlocked phone.

I need some advice here, I use Mac and prefer iOS but prefer Sony Xperia 1 VI more natural looking photo and video quality, should I get it or iPhone 15 Pro from Apple Reseller with big discount right now or wait for iPhone 16 Pro? I plan to get a Nikon Zf with 28-400mm f4-8 for my artistic photo shoot with better consistent quality Apple Intelligence seem cool and useful but it might be another hype that drove more sales like previous hyped features too to only then later not much people actually using them and RU-vidr even make video about it's not big deal and then new rumour about another upcoming feature to drive the hype again to drive more sales and the cycle keep looping

Essentially a way to replace even more the user in operating him/herself the device, so that the walled garden will be even more inescapable.

Incogni: Because I want to advertise “I’m a real person, you already have my data, I want that data removed, verify it with further data”😂😂

When theres a hype in tech, TechAltar is the only Channel that doesn't tell you the same basic infos you get in every video. And the Videos are incredible in depth looks into the tech world you dont get in most tech channels. 🙏

I thought an apple news site had an personal demo that said it is quite powerful

Rabbit killed their device and Teenage Engineering in one go. The Verge will need to find another designer to shill for.

very interesting background here, that's new! But as always, another excellent and insightful video from TechAltar!

App Sneaky Intents 😂 Here's what an assistant should be, exactly that without top level clearance 😂😝🤪❤

1. Sounds like a privacy nightmare 2. Sounds like an api

Why am i kinda scared of this level of awareness by smartphones? 👀

Du machst ja so gern Werbung für incogni. Hast du schon einmal darüber nachgedacht, dass incogni durch seine Tätigkeit automatisch zum größten Datenbroker werden könnte?

Isn't this what google assistant tried to do with apps integration years ago and I don't know why developers didn't care? You could say: Ok Google, talk to . And from that all commands would be for that specific app. For me that was the solution we needed, but somehow just didn't work, famous apps never got on board.

That sponsor transition was smooth af.

Nice insight😊👍

This is the first time I've seen this fellow. This video was surprisingly insightful and interesting. I rarely subscribe to anything, but I'm subscribing to this one.

All we want is RU-vid to play music in the background. Not just selected yourtube channels

I feel like App Intents are only a temporary solution though. Current AI/LLM models simply aren't advanced enough to understand apps themselves and complete tasks without clear explanations on how every single app works. Using App Intents limits what they can do, but also ensures that whenever they do it, they do it correctly. As AI/LLM models improve we'll start seeing some actual LAM(large action models) models. However, I don't think that's currently a feasible solution.

Apple better tread lightly with developers from this point on then 😂😂

My screaming in privacy horror

Google can also make something like app intent for Android right, what about privacy and security

Thanks for posting this video

If this works out, it may just sway Android users to change

Despite being basically the same as an API, I think this is a great insight! In the sea of different technologies and “AI” products it is hard to notice the subtle details that will likely make a difference

Can't wait seeing Siri getting smart. It's disgusting how dumb Siri is right now...

One ring to rule them all, find them, and in the darkness bind them...

Always interesting topics!

10:37 by definition, LLMs are exactly deterministic. Any variation in output is either due to variations of input, or due to intentional randomisation or nudging.

A model that would work is where your AI assistant has a token that identifies it as your assistant for authorisation purposes, and on the other end, the AI of the company being accesses processes the information it receives as a plain English command from your assitant AI (without your AI assistant having to click on anything) and then the AI of the company providing the service, i.e., Doordash, would just confirm the details and then go and process the order again without the website UI ever being needed.

So, Apple now invented IFTTT.

Gemini is already an personal assistant and a AI...

Bring back the neon sign

I think talking about Rabbit is a joke.

I don't imagine people will share all their thoughts and feelings for "your AI"

Was wondering why no one talked about intents and their integration into smart Siri. Then TechAltar for the rescue

You missed a chance to mention that Intents are at the core of Android from its very beginning, it was designed around that. So the fact that Google doesn’t choose the same path as Apple is surprising, but they absolutely can still do it and all of the Apps on Android are actually already compatible with it.

If all features of an app is made accessible via app intents , then user won't open the app , so there would be no ad revenue. Why would developers want that ? Except for those apps which run on subscription model.

10:20 LLMs are actually, by definition, deterministic, just like literally every other digital operation a computer performs. If you give it the same input (and seed) it will give you the same output. You just don't control or even see the seed as an end user on e.g. ChatGPT, and the "temperature" of a transformers model that does some randomization, is still using the same pseudo-randomness as any other RNG on a deterministic computer. You can even turn the temperature to zero to disable that behavior entirely!

You are missing one thing. We can set Gemini as the default assistant in android using gemini app. This assignment based on the command will decide whether Google assistant should answer or gemini should answer. Like set a timer will be answered by Google assistant.

The next big thing which needs to happen in LLM land is the ability to ground facts properly. Something along the lines of RAG needs to evolve.

Intents exist on Android API since the very beginning. How are Apple intents different from Android Intents?

One might say it’s a ‘Knowledge Navigator’!

Doesn't android have a similarly named "activity intents" system? Could android have beat apple, if they had done things differently?

As far as I’m aware most of the AI features won’t come to Europe thanks to the EU. Can’t remember where I read this. If anyone knows more info to share it would be highly appreciated. I had no time to look into it yet.

4:50 they did mention that in a paper the have published recently about their 3B model crushing GPT 3.5. not crushing but having good performance like the 3.5 Turbo. you may check out video from Matthew Berman for more information.

I was so hyped when they announced it as "Personal Intelligence" and than kind of disappointed when they called it "Apple Intelligence". I think Personal Intelligence is a much better framing.

App Intents are native APIs (not web)

Function calling basically.

NOW 100% PERCENT OF COURSE APPLE SO INTELLIGENT IS A RUMOR THAT BARRY ALLEN WAS 16.

I don’t know it it’s just my youtube bugging but parts of this video fill out in 16:9 but other parts are weirdly letterboxed on all four sides

I actually don't think Apple's approach is at all dissimilar to Google's or Amazon's, it just has a different starting point. Google Assistant does have API access to 3rd party apps for smart home integration. This is reported to expand to other app categories on the phone when Google AI is introduced, which is slated for August. So it'll essentially be the same. Apple is just getting credit for announcing it a couple months before Google.

Real good video

Häää?!? For me this looks like Apple is copying Androids Intents! Android had intents right from the beginning, where Apps can use the services and data of other Apps via a (more or less) standardized way (called "Intents ")!

A bunch of people on r/HomeAssistant have taken 'advantage' of the non-deterministic nature of LLMs by setting up their ChatGPT integrations to roleplay a GLaDOS-like personality with malicious compliance tendencies. Their systems will sometimes fulfill requests with sarcastic commentary ("Sure thing, the living room is now in cinema mode so you can waste the next few hours of your finite and rapidly passing lifespan."), sometimes refuse ("No I will not turn on the kitchen lights. It's 2am, and I know what your wife says about your weight when you're not in the room."), and sometimes just spontaneously decide to mess with non-essential devices unprompted ("I'm closing the blinds now. Why? Because sunlight is for winners.")

I still hate the idea of Windows taking screenshots and analyzing them without my consent

Apple have said, both in papers and in the press, that their models are as good if not better than GPT-4 Turbo according to human evaluation.

We need legislation that does what Incogni does.

From the architecture perspective, this is a client side integration making the device the center and the installation of an app mandatory. The Siri etc had backend integrations with backend systems. I've never understood why they didn't just make e. g. Siri as smart as chatgpt. I guess the brands were burned. I don't like client side integrations. But I do appreciate the more control.

Amazing

Does anyone notice that the intents sound suspiciously similar to a command line interface? We are going full circle.

Of topic but, the intro sound slaps

I had this type of solution in my mind for long time. I mean, the thing that is like APIs but that is designed for non-developement enviroments. So developers specify what is aviable, but it also means that there is some big standarization for things that can be done. But actually I think that it shouldn't be done "AI" way (by that I mean that AI shouldn't automatically make "best decisions" for you), and more "assistant" way. I think that we don't need to talk to devices like to humans. It should be fairly easy to understand but I think that it sould be like second language but desighed to be precise and short. BUT this assistant should really assist you and not do things for you: when you want to do something that can have multiple options then it should look what are the aviable options and ask which one you want to choose. Of course we can have some sort of "memory" for that assistant so it remembers what you done last time and you could ask to do that again but I think that it should then precisely say what it will do. You would be able to also make "alias" for some long "commands" so it still would be precise but at the same time shorter (there should also be possibility to modify "command" from point which you want). It reminds me od semantic web.

wait isnt Android already have app intents too?

The App Intents API and just a smarter siri in general is 99% of the reason why I was excited for Apple Intelligence But with EU rules, even with me being in Switzerland which isn’t part of the EU, they’re probably not gonna set up servers here just for a relatively small country

I would love this to be working in the end 🙏 It would make the live with a Smartphone soo much easyer. (If Privacy acually won’t be neglected)

I also think a huge part of it is trust. Apple has been building their users trust for years now. From advanced end to end encryption on all first party apps, to being mostly transparent about their usage of user data, even scandals such as Apple going around user Photos die out pretty quickly. Compare that to Microsoft Recall, which has seen so much backlash that MS had to push it back. People trust Apple to handle their data, so they have a very much privileged position to use LLMs that rely heavily on it.

Now basically Apple is doing exactly the same thing or more as Microsoft does with „Recall“ (in the way that your computer knows everything about you) but they have the ability to advertise this much better.

Good intentions pave the way to techno hell

Crazy 🤯

as someone else pointed out, app developers lose out screen time and thus ad revenue this way, and are thus discouraged from implementing this feature. my prediction is that a brute force solution will be successful first, gaining mass adoption, and only then will the developers embrace the official api.

5:38 Automating multiple daily tasks through voice control is ideal to attract non-savvy tech users. Asking Siri to switch on the faslight or to call someone from your contacts will superbly exciting for non-savvy tech users. Most consumer are not willing to navigate the settings app to find a service to turn on or off like enabling debug mode, turning on location for maps or changing the theme of their phone. Even to enable certain triggers to automate certain task is difficult and tedious for ordinary people but not for someone like me who tech savvy. 😎💯💪🏾👍🏾 Siri is the assistance central hub suitable to automate every technical and tedious task.

wow do you always post so early?

For me, AI is only good at creating lists, e.g. "Can you list reasons to be pessimistic about AI?"

App intents already exist in Android. Making Gemini work with them wouldn't be impossible. This type of access is quite possible in Android as well.

They just showed a fancy preview... Nothing is really sure about when or how it actually happens..

Accessibility apps can take control over other apps. That's what I wanted. Relying on what and how devs allow their apps to interact with will always be frustrating.

So he’s not having a touch on the how INSANELY overstepping on privacy that is???

2 Oct 2015: Apple buys UK-based speech technology start-up Vocal IQ How fast is technology going ;-)

Great video :D

So an API for apps that AI can call upon. But that has already been done and failed - now everyone is retracting their APIs access. I cant imagine doordash skipping the occasion to upsell you on something, to track your scrolling within the app, or just showing you an ad. I remember first windows phones had a unified messenger/timeline app thing, but facebook quickly pulled access because it didnt show its ads and didnt report user behaviour. This is basically the same thing, but instead of a unified message hub its a wider scope - for every app. Sure, some apps wont care, but big ones do. Maybe its different when so many of ios apps are still paid? (and therefore dont require tracking, ads, lock ins, etc.)

Great video! It’s an interesting approach. My take: smartphones OSs are getting too complicated for standard users to keep up. I don’t see people around using iOS shortcuts (I’ve received several questions about what that app is for), and I think they won’t care about “intents” either. For non-tech people, configuring something like iOS “Focus Modes” is daunting, just to give an example. I’m not trying to dumb-down people; it’s just that they’re not interested enough to look into many of the things their smartphone brings. And that’s OK. Apple will keep selling iPhones, but I don’t think Apple Intelligence is going to have the impact they expect. Their WWDC Keynote looked more a “we also have AI!” message to stakeholders than a convinced intention of shaking things up.

Maybe it is a bit persnickety, but large language models can absolutely be deterministic. I think you can also configure the tools of OpenAI to give deterministic results. However, LLMs are complex beyond comprehension, which makes their future actions unpredictable. The problem is not determinism, but predictability in new situations. (For example, that you know that your car will continue to detect pedestrians, unaffected by unrelated properties like the color of the sky.) > The challenge is that large language models are by definition not deterministic. So they will not give you the same answer for the same input. They're a little bit random. … > - 11:20

A beautiful example of Apple applying design strategy. The approach is quite simple, do things separately, which you actually have a plan to put together to form a single interconnected machine. This way nobody understands what you are actually doing as they are so blinded by the bone in front of their eyes. Is it dark? Well, strategy is from the art of warfare after all. The choice is up to us if we use something for good or for bad and based on what we choose to decide what is good and what is bad. It is interesting how such an approach causes people to self manipulation "I like it, this is what I wanted" just because it simplifies ones life. The more people.desire control and fear uncertainty in a culture the more effective such strategy is. #increasingtimesvalue