How Well Does Chat GPT Know Commander Cards?  

Didn’t notice that!

I also saw chatgpt using I.

Honestly for me it was pretty obvious and that gave it away a lot

Yeah AI don’t refer to themselves very often… in formal writing to drive a point home you don’t use the word “I.” I picked up on that too, lol.

That threw me for a loop as well!

Smothering tithe really was the hero of this story

Yea those are the AI hallucinations which causes it to be wrong

I made it anonymous so unless they tell me, I also won’t know more

I would have given it an A, but yeah, it isn't an S imo

I feel like that’s how a lot of AI work looks.

I swear they didn't used to and now they always do it, in any context. Its very frustrating.

it's got the vibe of trying to fill space in a high school writing assignment you reeeeally don't want to do

Yea I didn’t know it was gonna do that. Made for a funny bit though

Yea it was such a weird response for that card

Novel reasoning ends up being a huge giveaway! I think you are spot on

While watching (my guesses of the identities and tracking the rating scores). My guess for the AI in brackets. Actual AI in parentheses. 1 [(A)] or C. Initially guessed based on accuracy. Doubled down based on generic AI answer vs novel Patron answer. 2 [(C)] or C. Again, novel responses help guess the Patron. 3 [(A)] or S. One answer repeated itself in a redundantly redundant explanation. Huh, the AI downgraded it to nonland permanent. Was that due to generalizing answers or due to not understanding the semantics of the card? Both are factors but I wonder which had a bigger cause. 4 D or [(C)]. Initially guessed based on accuracy. Further confirmed by the novel Patron answer (silver bullet draft design). Even further confirmed by the LLM having no context for the lack of horsemanship. 5 [(S)] or B. Initially guess based on accuracy (unless the patron is trolling, or arguing that it is too powerful to fit in many commander decks without moving the deck away from the desired power level). Wow the reasoning is making me reconsider. The S ranking said "any deck within it's (Demonic Tutor's) colors (plural)". Why the implication of plural? There is also more redundancy in the S's reasoning. I am changing my mind. 6 [(A)] or B. The LLM likes listing literally the same logic repeatedly. The Patron response was more novel. 7 D or [(D)]. This one is tough. The left was more novel. Wow. I expected something like 55/45 odds there. Let's Go! 8 [(S)] or B. I initially guessed based on accuracy, but the B has the novel response, so it must be the Patron. LLM wouldn't do that. And once again the LLM uses "decks within it's colors" when talking about a mono white card. Why the plural? Also the card needs to fit within the deck's colors not the deck fit within the card's colors. 9 [(A)] or A. "Decks focused on defending against large attacks"? Also the Patron is once again the more novel answer. 10 S or [(S)]. Redundant LLM response is redundant.

After the 10 scores: Patron scores: SSABBBCCDD (5 different ranks. Somewhat biased towards B but really spread out otherwise) LLM's scores: SSSAAAACCD (4 different ranks. High bias towards S or A) Since my 10/10 accuracy was based on my reasoning of the LLM's limitations, I think it is soft evidence that my predictions about its limitations might be accurate.

Bonus Round? 1. [(B)] or C. The C had a novel response. Final thoughts: We already know ChatGPT does not try to evaluate cards, so it is not suited to evaluating cards. (Don't use a saw for a hammer's job). Beyond its lack of motivation to judge cards, it does not understand the card or their context enough to judge them. Additionally we see it's general answers as a clear marker of the LLM answer. It is trained to give a "reply-like" response that was a likely reply rather than a reply that was likely to be correct. Specificity and nuance are things it is trained to avoid.

Your patron's evaluation seems within the norm for commander players. They can mostly evaluate cards, and there is some subjectivity that make the "surprising" evaluations still have merit.

That’s funny maybe I’ll need to try that too

Valid point, especially in casual settings and for decks with a very clear and not super varied gameplan. For example, my Kathril deck only really wants to fill the graveyard with keywords, and i took Entomb out of it because i would always tutor up Zetalpa which made the deck play very monotonous (amplified by how terrible the precon is at filling its graveyard so i took a lot of mulligans, but Entomb of course was always keepable) and now that i'm replacing a ton of cards soon, i think i might also cut Vile Entomber and Buried Alive, and exclusively rely on what i happen to mill/sacrifice.

I do like this idea, though the flexibility of a tutor I think makes it worth it!

What he said the flexibility to get let's say a board wipe OR a single target removal spell because you have a big board presence makes it way better imo

@@Jacob-km4yb . . . I know

I’m glad! I know this one is a bit different so I’m happy you like it

Interesting I often think of it as a charm effect. I don’t know if I’ve ever been explicitly targeted because of it

Me making articles or videos feeds it data. Not me asking questions. Just asking questions isn’t really training it

I would not be surprised if the questions are saved as more raw data to feed it later.