Installation
LLM Evaluators
Battle
Compare if a solution performs better than a reference solution.ClosedQA
Evaluate answer correctness using the model’s knowledge.Factuality
Check factual accuracy against a reference.Humor
Rate the humor level in text.LLMClassifier
High-level classifier for evaluating text using LLMs.Any
required
Any
required
Any
required
Any
Any
Any
Any
Any
Any
Any
Optional[Client]
Possible
Evaluate if a solution is feasible and practical.Security
Evaluate if a solution has security vulnerabilities.Sql
Compare if two SQL queries are equivalent.Summary
Evaluate text summarization quality.Translation
Evaluate translation quality.String Evaluators
EmbeddingSimilarity
String similarity scorer using embeddings.Any
Any
Any
Any
Any
Optional[LLMClient]
ExactMatch
A scorer that tests for exact equality between values.Levenshtein
String similarity scorer using edit distance.Numeric Evaluators
NumericDiff
Numeric similarity scorer using normalized difference.JSON Evaluators
JSONDiff
Compare JSON objects for structural and content similarity.Scorer
Scorer
bool
ValidJSON
Validate if a string is valid JSON and optionally matches a schema.Any
List Evaluators
ListContains
A scorer that semantically evaluates the overlap between two lists of strings. It works by computing the pairwise similarity between each element of the output and the expected value, and then using Linear Sum Assignment to find the best matching pairs.Any
Any
RAGAS Evaluators
AnswerCorrectness
Evaluates how correct the generated answer is compared to the expected answer.Any
Any
Any
Any
Any
Optional[Client]
AnswerRelevancy
Evaluates how relevant the generated answer is to the input question.Any
Any
Any
Any
Optional[Client]
AnswerSimilarity
Evaluates how semantically similar the generated answer is to the expected answer.Any
Any
Optional[Client]
ContextEntityRecall
Measures how well the context contains the entities mentioned in the expected answer.Any
Any
Optional[Client]
ContextPrecision
Measures how precise and focused the context is for answering the question.Any
Any
Optional[Client]
ContextRecall
Measures how well the context supports the expected answer.Any
Any
Optional[Client]
ContextRelevancy
Evaluates how relevant the context is to the input question.Any
Any
Optional[Client]
Faithfulness
Evaluates if the generated answer is faithful to the given context.Any
Optional[Client]
Moderation
Moderation
A scorer that evaluates if AI responses contain inappropriate or unsafe content.Any
Any
Any
Optional[Client]