DeepSmith

Nov 26 · AEO & AI Visibility

18 min read

How AI Sentiment Analysis Works and Where It Breaks

Avinash Saurabh
Avinash Saurabh · CO-Founder & CEO
A semicircular sentiment gauge with a needle near the middle, surrounded by layered answer cards joined by thin node lines, under the words Reading Sentiment Scores Carefully.

AI sentiment analysis reads a piece of text and classifies the attitude it expresses as positive, negative, or neutral, sometimes with a score or a confidence value, and sometimes tied to a specific brand or feature. If you use AI visibility tools to track how AI engines talk about your company, you've probably seen a sentiment number on a dashboard and wondered how much to trust it. It's a fair question, because that number is an interpretation of an answer, and interpretations can go wrong in some predictable ways.

This guide walks through how a sentiment score gets made, why the same answer can look positive, negative, or neutral depending on the tool, and the places where scoring tends to fail, such as hedged wording, mixed claims, and sarcasm. By the end you should be able to look at a score and know what questions to ask before you act on it.

What does AI sentiment analysis actually measure?

It measures the evaluative attitude expressed in text, which is a little different from whether the text mentions something good or bad. The output can be a label (positive, negative, neutral, and in some systems mixed), a number on a scale, or a set of opinions that are each linked to a target. Which of those you get depends on the system.

One thing to get clear early is where the score comes from. An AI engine like ChatGPT or Perplexity generates an answer. A separate sentiment system, or a step inside a monitoring tool, then reads that answer and assigns the sentiment. There's no single standard score attached to every AI answer. Each provider or product picks its own labels, scale, sampling, and way of adding things up, so one tool's number isn't directly comparable to another's unless you know their methods match.

A few terms get mixed up a lot, so it helps to pull them apart.

  • Polarity or label. The direction of the sentiment, usually positive, negative, or neutral.
  • Score. A number whose meaning depends on that system's scale. Google Cloud Natural Language, for example, documents a score from -1.0 (negative) to +1.0 (positive).
  • Magnitude or intensity. How strong the emotional expression is, which isn't the same as its direction. In Google's version, magnitude starts at 0.0 and has no upper limit for a document, and longer documents can pile up more of it.
  • Confidence. How strongly a classifier favors the label it picked. Microsoft's language service reports confidence for positive, neutral, and negative as values from 0 to 1 that add up to 1. A very confident "negative" is still just negative. It doesn't tell you the text is strongly negative.
  • Aspect or target. What the opinion is about. A brand can be praised for reliability and criticized for price in the same answer.

Keeping these separate saves you from a common misreading. A score near zero might mean the answer had very little emotion in it, or it might mean strong praise and strong criticism cancelled each other out. In Google's documentation, a mixed example has a score of 0.0 and a magnitude of 4.0, while a neutral example has a score of 0.1 and a magnitude of 0.0. Same score, very different text. And a high confidence value doesn't show that the interpretation is correct, only that the classifier favored it.

How does a sentiment score get made?

Most systems follow a loose pipeline. They find the relevant text, work out what it's about, interpret the sentiment-bearing language in context, assign a label or number, and then combine sentence-level or aspect-level results if someone wants one result for the whole document. The details differ by tool, and a dashboard in one of the LLM monitoring tools you use may add its own counting rules on top.

Lexicons and rules

The oldest approach is a lexicon, which is a list of words with sentiment values attached, plus rules that adjust for context. VADER is a well-known example built with social media language in mind. Its lexicon has more than 7,500 validated token features, drawn from a larger pool of more than 9,000 rated features. It adds up the word values, adjusts them with rules, and squeezes the result into a "compound" score from -1 to +1.

The rules handle things like negation, degree words ("very"), word order, capitalization, punctuation, emoticons, and contrast words like "but." VADER's published thresholds call a compound score of 0.05 or above positive, anything between -0.05 and 0.05 neutral, and -0.05 or below negative. Those cut-offs are VADER's own conventions. Another tool can draw the lines somewhere else, and neither is wrong, which is why you can't carry a threshold from one product to another.

Trained classifiers and large language models

A supervised classifier learns from examples that people have already labeled. Its behavior depends on the examples, the domain, and the task it was trained for, and a model trained on one kind of text may not carry over well to another. A large language model can also be prompted to label sentiment. The answer will read smoothly, but a smooth answer doesn't guarantee the polarity is right or that the model worked out correctly who said what about which feature.

The task makes a big difference here. Labeling a short review as mostly positive is easier than pulling out every opinion, its target, and its aspect from a long answer that makes many claims. A system can get the first job right and still pin a criticism on the wrong brand, or miss one negative point tucked inside a mostly friendly answer.

How results get combined

Once you have sentence-level labels, someone has to decide how to roll them up, and that choice changes the story. Microsoft's documentation describes one rule: a document counts as mixed when it has at least one positive and at least one negative sentence. If every sentence is neutral, the document is neutral, and a positive sentence next to only neutral ones gives a positive document. Google's approach is different, since it relies on magnitude to help you tell low-emotion text from text where opposing feelings offset each other. Not every tool has a magnitude field, and not every tool uses the same mixed-label rule, so check what yours does before you compare numbers.

Why can one answer look positive, negative, mixed, or neutral?

Because the overall tone and the opinion about each feature are two different questions. Overall sentiment asks what the general mood of the text is. Aspect-based sentiment asks what the opinion is about each target. Take "The restaurant was expensive, but the menu was great." An aspect-based analysis can attach a negative to the price and a positive to the food. A single document label has to squash those into one answer.

A fuller way to describe an opinion has several parts: the entity, the aspect, the polarity, who holds the opinion, and when it was expressed. Each part protects you from a specific mistake.

  • The wrong target. A nice comment about a competitor isn't favorable sentiment toward the brand you're measuring.
  • The wrong aspect. Criticism of pricing doesn't necessarily mean criticism of reliability or overall usefulness.
  • The wrong speaker. A generated answer might say that reviewers disliked a product without endorsing that view itself. Sentiment inside a reported opinion and sentiment in the answer's own conclusion are separate things.
  • The wrong time. An answer may describe an old problem and then say it was fixed. If a tool flattens both statements into one, it can misstate the current picture.

Here's a made-up example to show how this plays out. Suppose an answer says, "Reviewers found the battery reliable, but its own conclusion is that the price is high." A good reading keeps the reviewers' praise apart from the answer's judgment about price. A rough reading might average them into a mild positive and move on.

It also helps to know that sentiment isn't the same as stance. Sentiment is the positive, negative, or neutral tone of the text. Stance is the author's favorability toward a specified target, and it can be implied without the target being named. Research on stance often uses an example where a tweet praises one political candidate and, by implication, opposes another. For brand work, the lesson is that being mentioned or cited doesn't tell you the sentiment, and you shouldn't infer how an answer feels about your brand from a contrast with a competitor. Read what it says about you.

Where does sentiment analysis break?

The failure modes below show up again and again in the research. The numbers attached to them come from specific studies on specific systems and datasets, so treat them as evidence that a weakness exists, and not as error rates for any tool you use today.

Negation and its scope

Negation can flip, soften, or redirect a sentiment word, and where it applies isn't always obvious. "Not poor" doesn't mean the same thing as "poor," and in a longer sentence the "not" might cover only one clause. In 2020, the CheckList study behaviorally tested commercial APIs and research models and found failures on negated-negative sentences like "The food is not poor." Across the five systems it tested, failure rates on that kind of example were 18.8%, 54.2%, 29.4%, 13.2%, and 2.6%. Those figures come from one targeted test, so they don't describe how often these systems are wrong in general.

A 2016 study on review text treated negation and speculation as two linked jobs, finding the cue word and finding how far its scope reaches. Its method improved on its baseline by roughly 20% in negation-cue detection and roughly 13% in scope recognition. That's useful mostly as a reminder that scope is a hard part of the problem in its own right.

Hedges and uncertainty

Words like "may," "might," "appears," and "possibly" change how certain a claim is. A system can notice a sentiment word, miss the hedge around it, and score a cautious possibility as a firm endorsement or a firm criticism. Take "It may be cheaper, although reliability is still untested." That sentence has uncertainty and two separate aspects in it, and a single positive or negative label throws most of that away.

AI answers hedge a lot, so this one matters for AI brand monitoring. One caution about what the research does and doesn't say: the 2016 review-domain study uses the term "speculative information," and no standard hedge score has turned up or universal rule for adjusting for hedges. Don't assume your tool has any hedge handling unless its documentation says so.

Mixed claims

An answer can say good things and bad things about different aspects, or even about the same aspect. One polarity score hides that mixture. A near-zero result is the clearest case, because it can mean "not much said" or "a lot said, and it cancels out." When the difference matters to a decision, go to the answer text and any aspect-level labels instead of relying on the bare total.

Sarcasm and irony

Sarcasm often uses positive words to deliver criticism, so the literal polarity of the words points the wrong way. Picking it up can depend on context the classifier doesn't have, such as the author's intent, an earlier conversation, or cultural cues.

The iSarcasm dataset, published in 2020, has 4,484 tweets that their own authors labeled and explained. It separates intended sarcasm from perceived sarcasm, and the two don't line up well. In that study, 30% of the sarcastic tweets went unrecognized by annotators, and 45% of the tweets people perceived as sarcastic hadn't been meant that way. Agreement between annotators was modest, with Cohen's kappa values of 0.37, 0.39, and 0.36. The best model scored an F-score of 0.364, against 0.616 for human annotation. These results apply to that dataset and task. Still, if trained humans struggled to agree, it's reasonable to expect that a scoring system will sometimes get irony wrong too. AI engines don't write much sarcasm themselves, but they do quote and summarize text from forums and reviews where it's common.

Context, domain, and bias

Words depend on where they're used. "Long" is good for battery life and bad for a waiting time. A model trained on one domain can do worse in another.

Bias is a separate concern. A 2018 evaluation ran 219 sentiment systems against an 8,640-sentence English corpus built to look for race and gender bias, and several systems consistently gave slightly higher sentiment-intensity predictions for one race or gender. That shows bias can appear in tested systems, not that every system behaves the same way. A 2025 study on polarity association found that models may link certain words or situations with positive or negative polarity and then misjudge neutral statements. Its authors note that their correlations don't prove that mentioning a word causes the error, so read it as a possible failure mode and not a settled fact about all language models.

Prompt and model variability

When a language model does the labeling, the exact instructions and output format can change the result, especially for structured, fine-grained tasks. A 2024 study found strong zero-shot results on some simple positive-or-negative tasks and weaker results on aspect-based and implicit sentiment. In its tests, one model reached about 54% of a fine-tuned T5 model's performance on aspect-based analysis, and 45.25% accuracy on implicit sentiment against 67.12% for the fine-tuned model. The same study found that structured tasks were sensitive to how the prompt was written. Those results apply to the models and datasets it tested.

Confidence needs its own warning. Calibration research asks whether a model's probability estimates match how often it's actually right, and a well-known 2017 study found modern neural networks were poorly calibrated in the settings it examined. So a confidence value isn't automatically a trustworthy probability of being correct. That study didn't test today's sentiment products, so use it as a reason to be careful, not as a measurement of any one tool.

How should marketers read brand sentiment in AI answers?

Start by remembering what the number is. Tracking your brand in AI answers produces several measures, and sentiment is only one of them. Brand sentiment in AI answers is an interpretation of the answers that one tool collected and analyzed. It isn't the same thing as how often you're mentioned, how often you're cited as a source, or your share of voice. Those are separate measures, and a brand can be mentioned a lot with lukewarm sentiment, or mentioned rarely with glowing sentiment.

It also depends on how the answers were collected. Ahrefs describes its Brand Radar method as querying supported public AI interfaces, storing the raw responses, and searching that collection for mentions and citations. Its questions come from People Also Ask data and a large keyword database, it favors common high-demand questions, and it leaves out some niche and long-tail ones because the number of possible AI prompts is effectively unlimited. That's one vendor's design, and its documented metrics cover mentions, citations, impressions, and share of voice, not a sentiment score. Semrush's AI Brand Performance documentation, in contrast, describes sentiment as favorable versus general and says the sentiment metrics rest on non-branded queries, with an answer panel where you can read the generated text. It doesn't publish a formula or a numeric range, so there's nothing you can safely assume about how its sentiment is calculated.

For sentiment analysis tools marketers are shortlisting, a roundup of brand monitoring tools for marketers shows what each one covers, though you'll still want to ask the method questions below. The point of those two examples isn't to rank anyone. It's that a sentiment figure reflects choices about prompts, platforms, and analysis that you can't see from the dashboard label alone. When you're comparing sentiment analysis tools marketers might use, the useful questions are about method: which model or rules produce the label, where the neutral line sits, how results are combined, and whether you can open the underlying answers. The same applies if you're weighing tools that score brand sentiment against each other.

DeepSmith tracks Sentiment as one of its metrics, showing whether AI describes your brand positively, neutrally, or negatively, next to Mention Rate, Citation Rate, and Share of Voice. It's best read the way you'd read any sentiment metric, as a prompt to go look at the answers behind it.

A checklist for reading a score

Here's a short routine you can use on any sentiment number you're about to share.

  1. Read the actual AI answer before you interpret its score.
  2. Confirm which brand or entity the statement is about and who is expressing the opinion.
  3. Separate feature-level judgments, like price, reliability, and support, wherever the tool lets you.
  4. Look for negation, hedging, contrast, time references, and sarcasm before you accept a label.
  5. Check whether "neutral" means low emotion, a default category, or an average that hides mixed feelings in that tool.
  6. Keep confidence, intensity, and polarity apart, and check how the tool itself defines each one.
  7. Compare only results collected with similar prompts, platforms, locations, time windows, and analysis rules, and check whether you can export samples and answer history.
  8. Describe a change in score as a change in the measured sample, which isn't proof that brand perception shifted for everyone or that a particular event caused it.

Step 8 is the one people skip most. A move from 0.2 to 0.1 might be real, or it might come from a different mix of prompts that week. If you're deciding which numbers belong on a report, this guide to what a monitoring dashboard shows is a useful companion. If you want a way to separate real movement from noise, a guide on what counts as a meaningful change in AI visibility covers sample gates and persistence rules that apply here too.

What the evidence can't tell you

It's worth being honest about the limits. There's no universal sentiment formula or cross-vendor standard. Nothing in the research shows that sentiment in AI answers predicts what customers think, whether they'll buy, or how your reputation will fare. And the exact model, thresholds, aggregation, and training data behind any given product's sentiment metric have to come from that product's own documentation, since a dashboard label won't tell you. To test a vendor's numbers yourself, you can audit a tool's data quality by sampling prompts and comparing them with live answers.

That isn't a reason to ignore sentiment. Scores can summarize patterns, and a steady trend in a well-defined sample is useful. They just work best as a pointer to answers worth reading, not as a verdict. If you'd like to see how your own brand shows up in AI answers, including sentiment, you can start a free DeepSmith trial and read the answers behind each number.

Frequently asked questions

Is a sentiment score the same as confidence?

No. A sentiment score or label says which direction the text leans, and in some systems how strongly. Confidence says how strongly the classifier favors the label it chose. A system can be very confident that a sentence is mildly negative, or unsure about a sentence that is strongly emotional. Check how your tool defines each one.

Does a neutral result mean the answer is unbiased?

Not necessarily. Neutral can mean the text expressed little emotion, that the system fell back on a default category, or that positive and negative statements were averaged into the middle. A neutral label also says nothing about whether the answer is accurate or fair to your brand. Reading the answer is the only way to know which case you're in.

Can sentiment analysis detect sarcasm?

Sometimes, but not reliably. In the iSarcasm study, the best model scored 0.364 against 0.616 for human annotation, and people themselves often disagreed about what was sarcastic. Those numbers come from one dataset, so they aren't a rate for any particular tool, but they're a good reason to read answers that look surprisingly positive or negative.

Why do sentiment tools disagree with each other?

They make different choices about the model or rules, the scale, where neutral begins, how sentence results are combined, and which prompts and platforms get sampled. Two tools can read the same answer correctly under their own definitions and still report different numbers. Compare trends inside one tool, and be cautious about putting two tools' numbers side by side.