OmniEval is a new benchmark for evaluating Retrieval-Augmented Generation (RAG) systems in finance.
Key features include:
• Matrix-based Evaluation: Assesses five task types across 16 financial topics for detailed analysis.
• Data Generation: Combines GPT-4 automation with human checks, achieving an 87.47% acceptance rate.
• Multi-stage Evaluation: Measures both retrieval and generation performance for a full assessment.
• Robust Metrics: Uses rule-based and LLM-based metrics, enhanced by manual annotations and fine-tuning.
Experiments show OmniEval’s effectiveness in highlighting
#RAG system performance variations, offering insights for improvement in specialized domains.
The benchmark is open-source and available at https://lnkd.in/eFW5acgG.
For more details, read the full paper here:https://lnkd.in/e7yEGJ6e
#rag
#ai
