Stop Using the Wrong AI Model! Here's How to Choose Perfectly Every Time

for the past two years there has been one single question that I've been asked over and over and over and over and over again I get asked this question literally every single day it is by far the most common question in all of AI history and you yourself have probably asked it you've probably asked it several times and this question is really just an indication of a huge problem we have in the world of AI so what's that question what is that one question that I get every single day and

have answered probably a thousand different times and a thousand different ways it's this what AI model should I use in a world where there are literally hundreds of different AI models to choose from this is an understandable question when you are presented with a whole lot of options of something the question naturally becomes which one is the best one but the reality is the reason why this question is asked so much and why it hasn't been answered easily in the past is because it's a very difficult question to answer and in

this video I'm going to hopefully show you why it's a difficult question to answer and give you some guidance on how you might be able to answer that question for yourself now to emphasize why this is such a big problem and why you probably feel the uh anxiety and the tension of trying to answer that question for yourself there was a study study done decades ago where a research group they took two groups of people the first group of people were presented with a small number of jams uh they put out samples on a table and they allowed

people to come and Sample the different jams and see if they would buy one another group was given also a set of jams to choose from only this time they were given many more options a lot of options to choose from and the results of this experiment were pretty eye openening the group who saw only a handful a few of options ended up buying more frequently the group that were presented with more options actually bought less so the lesson learned here and the the concept now known as the Paradox of choice teaches us that when we are presented with an overwhelming

amount of options we tend to not choose at all hence why we have such a hard time with this question of which AI model should I choose and the AI companies aren making it easy for US Open AI has what 7 8 9 12 models I don't know I've lost count Claude anthropic the makers of Claude have what seven or so models themselves Gemini who knows how many models they've got and that's only scratching the surface there are dozens of AI companies fighting to be a name in the space and all of them having

different versions and different iterations and different approaches to their AI models now in Magi we try to make this a little bit easier for folks when you go to select a model in Magi yes we do have quite a few options to choose from but we don't have every option one of my jobs I believe as the Creator and founder of Magi is to not give users every option Under the Sun I want to give our users only the best of the best enough to cover the full spectrum of capabilities and costs that

are available on the market so Magi will never have have hundreds of different options in fact we probably have a few too many right now and I'll be going through and making some adjustments but even still there are over 20 different AI models I I think the last I counted was about 28 to choose from now we also try to make it easier in that we have an auto option now this auto option allows you to select it and then once you send your prompt the AI underneath the auto model will actually evaluate your prompt and based on a bunch of data that we've

collected will route it to the best llm for that particular prompt but even still it's always best to have a rich understanding of how these models work and what they're good and bad at so that you can eventually make the decision for yourself but how do you do that well one way to do that is through looking at evaluations of these models now every time a company comes out with a new AI model they typically have some charts and graphs saying oh our model excelled at this and look how much better it is at this than these other competitors

which is good I guess but how much can you actually trust those companies of course they're going to be biased and they're going to show only the Highlight Reel and not show you where their models were maybe not so good that's why it's great to have something like the chatbot Arena if you're not familiar with the chatbot Arena this is a place where AI models are put through a ridiculous amount of tests they are put through the ringer into a virtual Arena of sorts where they battle it out to see who is

the best of the best now these evaluations are thoughtfully done they cover a large amount of categories and Fields and verticals and as a user you can head over to the chatbot Arena over on huggingface face. and you can actually look through and see how every single model that's gone through the chatbot Arena lines up to its competitors and you can see they're scoring and you can see how many votes they got because these evaluations are not necessarily based on scores but they're also based on user voting so you get a combination of technical

evaluation and human evaluation and I think that's what makes it really beneficial now the other thing you can do here on chatbot arena is you can sort by category Now by default it's going to be set to overall so it's going to show you the the top ranking overall models and you can kind of sort through them or you can choose a category that makes the most sense for you for for me it's typically creative writing and you can see which models tend to excel in that area now even still these evaluations are extremely comprehensive and not only

that you have humans who are helping to upvote the ones that they think are better but it still Falls kind of short and here's why most of these evaluations are engineered around General Uses and every category they they might specialize in a particular category but the evaluations are still very generalized they may not reflect your specific use cases and some of these evaluations May overemphasize the flashy technical stuff over the actual practical usable stuff for the average

person so what I've done is I've put together a little bit simpler evaluation process how do we compare these models in a way that we can figure out which one is the best for our given scenarios now before I start going through my simplified methodology of finding the right AI model I want to give you a story analogy imagine you had a neighbor named John and John just got access to a fleet of new vehicles he inherited them from a relative and now he's got sitting

in his driveway a brand new electric vehicle state-of-the-art super eco-friendly and in pristine condition he's also got a Ford F350 truck super heavy duty souped up ready to haul the heaviest haul that you could possibly ever want to haul he's also got a Ferrari concept car with a top speed of 200 mph this thing is really nice and then he also has a golf cart which he can use to cart himself around now John

isn't very Savvy about how to use these different vehicles for their specific purposes he's a bit dense JN is and so over the weekend he decides to go golfing and takes his Ford F350 truck cross the fairways all over the greens and just ruins the club he's banned for life and then he has to go grocery shopping and decides oh the Ferrari is probably the one I want to take for that and takes his gas guzzling Ferrari down the street to the corner market to grab

some groceries and head back now obvious VI ly John doesn't quite understand the full capabilities of what he has on his hands and I think that's how many of us are actually using these AI models if we don't understand what they're capable of now for our intensive purposes we're going to break up llms into two categories we have the standard llms and we have the thinking llms or the reasoning llms and this is a recent advancement in large language models that kind of breaks them into these two

categories the standard llms were the ones that we started with you send your prompt and they answer quickly the reasoning or thinking models these models will first take your prompt and interpret it and they will attempt to Think Through the solution in a sense responding in a thoughtful process trying to break down your query and understand the best way to answer that query rather than just responding right away the result of this is they can

typically handle much more complicated tasks they get to Think Through very Advanced Solutions and tend to be a little more comprehensive in their responses now the thing with that is they tend to be a little more expensive obviously they have to take more processing power in order to complete the request thereby consuming more energy and thereby encountering more cost so step one in evaluating which model you should use should always start with what type of problem is it I'm trying to solve is it a highly complex

problem or task or is it a simple problem or task answering that question first you can immediately filter out the category of models that would be ideal or not ideal for your given task for the highly complex and highly Advanced and lots of nuance involved tasks you want to use the reasoning models for the tasks that are very simple maybe very straightforward and quick you want to

use the standard models in Magi we make this fairly simple we have little icons next to each AI model we offer and these icons represent different features or capabilities if you see a little brain icon on the model that means it is a reasoning model it is a thinking model so if your task requires in-depth thinking and reasoning and strategizing and deep knowledge and Nuance you can easily identify the thinking models right away now the second layer of how to choose the right language model is

what capabilities do you need do you need it to view an image do you need it to respond to an image then you obviously need a model that has a vision capability and in Magi these are showcased by the I icon uh to indicate that it is Vision enabled and lastly it do you need it to perform an action is there an outside tool that you need it to use in order to complete this task for those models we have the uh little code icon activated here to show you that his actions enabled now you might

say Dustin why wouldn't I just choose the most powerful model well we go back to our analogy of John our somewhat thickheaded neighbor you don't necessarily need a Ford F3 350 to drive across the golf course you don't necessarily need a race car to go grocery shopping and in the practical sense for you using an AI model the real question here is cost now the heavier more advanced more energy

greedy models they tend to cost a little bit more and even if you're using an app like claw or chat GPT you actually get limited use of these models so you really want to reserve the more complicated tasks for the more complicated models especially when it is a high value task that is worth the added cost whereas if you have a very simple task that doesn't require a whole lot of uh Nuance or or strategy or highlevel thinking you want to use a simpler model and not rack up your costs

or use up too much of your monthly quota of usage for a given model and that's why Magi again we also show you how much a model costs based on these usage multipliers so you can see the more advanced ones such as GPT 4.5 they cost 20 times as much as the Baseline llm whereas GPT 40 costs a 1x which is the Baseline so you can further help your decision-making process by figuring out number one how valuable is this task do

I want to spend a whole lot on it or does a simpler model with lower cost suffice now again one of the things that I would add here is that your specific use case might be much different and necessitate a different approach than I might take with that same exact task or category of task and so really what I tend to tell people what I think is the most beneficial is that you just explore take a little time and play with the different AI models and see how they

respond to everyday tasks that you might have them do this is literally the only way to understand which model is going to be best for your given tasks think of it this way if you've ever worked at a place where you've had co-workers after working with them for a while you might get to know what Bob is good at and maybe what Bob is not very reliable at and you might be able to learn what Susan is good at and the things that Susan may not enjoy so much or might not do a great job at as as you work with

co-workers you tend to learn their strengths and weaknesses over time and the only way you do that is by working with them actually interacting with them and the same is true with llms and and any AI model the more you try them the more you explore them the more you will understand what they're good at and what they're not good at and the benefit of using them inside of Magi is that you can play them against each other so you can say to one model write me a high coup about Batman you can evaluate how that model did with that Hi cou and then you can switch to a

different model you can rerun the generation and see how that model takes on that specific task and you can do this over and over again you can even have them go back and forth throughout the conversation having multiple AI models contribute to the same conversation so my highest recommendation for you is to explore these models give them some tasks that you need help with on a

day-to-day basis and see how different llms respond to them differently now I'm not going to leave you there I'm going to give you some practical advice from my own experience now having run an AI company for the last two years I've probably had tens of thousands of interactions at this point with different AI models so I have some opinions and I have some feelings about what these models are good at and what they are not good at based on my own experience and based on that experience I've come up with a pretty simple flowchart and this is a flowchart which llm should I use and I kind of start out

with the evaluation of how complex is the task right so I've got some examples here so for complex analysis or math if it's a legal document review something that is highly sensitive and highly nuanced if it is scientific research synthesis and I would even add further if it requires multi-step strategic elements then I would go to the reasoning models that's how I would sort of filter out all of the other standard llms but if

it's something like content creation or customer support answering simple questions simple data extraction or structuring or summarization I would probably go with the standard llms now from there once we know whether we're doing the reasoning models or the standard llms there's really three specific categories of things that I typically filter through and that is does this task require creative thinking or ideation does it require accuracy

empathy or copyrighting does it require structured verbose or technical details based on those three C categories I then have my selections if it requires creative thinking or ideation I typically go with o1 from the reasoning model side 01 is open ai's most advanced reasoning model it is very thoughtful and highly creative and I found it to be really really good at brainstorming coming up with new ideas or taking new spins on ideas that I might have uh I

really love it for coming up with analogies with high levels of new on and uh even coming up with highly strategic initiatives that require multiple steps and multiple layers of strategy deep strategy when it comes to accuracy empathy or copywriting I pretty much always go with 3.7 sonnet in this case it is my favorite model to work with at this time uh 3.7 sonnet is anthropics newest thinking model it is actually really good at pretty much everything uh

but what this model does that the other models don't do is it can actually decide whether or not it needs to go to a thinking process or if it needs to respond straight out and it's very good at making that decision based on the user's query um and again it is by far in my opinion it writes the best language for uh human empathy and it sounds the most humanlike I haven't found any model that I've evaluated that just writes better than than Claude does and you you might find that to be different for you you might have a different approach and you might have a

different writing style um but for me Cloud 3.7 is the winner when it comes to accuracy empathy and copywriting now if I need something that is highly structured and I want it to be ver verbose and highly technical I will go with the Gemini 2.0 thinking model Gemini uh created by Google is a very technical very verbose model by default now you can of course prompt it to be less veros and uh you know a little more concise but out of the gate I found that it does tend to be a little more verbose and but it does really well with highly

structured stuff so articles U formulating outlines and that kind of thing so Gemini 2.0 would be my go-to for those types of tasks now on the standard llm side when it comes to creative thinking or ideation I found that I really like GPT 40 even though GPT 4.5 is out the cost of GPT 4.5 is just ridiculous this it is at a 20x multiplier it is in in some aspects 30 times more expensive than its

predecessor gbt 40 um and I found it's not that much better at least not enough to justify the cost I found GPT 4 is still very very good at standard creative thinking or ideation tasks now for uh accuracy empathy and copywriting again I'm going to go with Claude here you'll see a pattern with all of these uh gpts tend to be the creative thinkers Claude tends to be the humanlike writing and empathy and then Gemini tends to be the structured so um Claude 3.5 son is my

go-to for the humanlike writing and then Gemini Pro is my standard model that I go to for structured verbose and Technical outputs so that's my simple yet not so simple answer to the question which llm is the right one for my given task um I hope it's been helpful again the most important thing is that you figure out which one works for your specific needs and you have that freedom to explore all the different options inside of Magi in one beautiful interface remember to start with whether

or not the task is simple or complex and requires a lot of thinking and strategizing and then from there based on the capabilities and strengths of the model you can make your selection if you need any further help on this my team and I would love to help answer your question questions and work with you to figure out which one is the best for your given needs I hope that helps and we'll see you soon