Skip to content

Machine Translation Quality Estimation (MTQE)

Check out our blog post: Quality Estimation in Machine Translation: A Comprehensive Analysis.

Click here to view the API documentation.

What is it?

Pangeanic's MTQE (Machine Translation Quality Estimation) system is a patented inference framework, based on Artificial Intelligence (AI) techniques, designed to evaluate the quality of text generated by Machine Translation (MT) without requiring the use of human reference texts (gold standard translations).

  • Low Latency: Operates with low latency requirements for real-time prediction of the utility and perceived quality of MT output.
  • Integration: Provides integration support for various production environments and localization workflows.
  • API Access: Access is facilitated through an application programming interface (API), allowing implementation in automated, high-performance translation pipelines.

Methodologies and Key Features

MTQE relies on an exhaustive set of linguistic and statistical features, that are detailed below. It uses pre-trained and finetuned models, along with internal mathematical operations, to generate quantitative quality predictions. It functions as a black-box evaluator, requiring only the source text segment and the segment translated by MT. This architecture ensures compatibility with any third-party MT engine.

Developed on advanced language processing architecture, the system has been calibrated with a large volume of hybrid data, blending human validations (Direct Assessment and MQM) with artificial data. This empowers the model to score translations addressing factors such as:

  • Content and Coherence Failures: Cases where the translation completely fails to communicate the original message (incoherent text, fluency without semantic relation, or absence of translation).
  • Structural and Integrity Defects: Information omission, redundancy (duplicate text), and defective punctuation.

Coverage and Adaptation

  • Multilingual Support: The system handles a training corpus supporting over 50 language pairs, covering high and low-resource environments, including languages with European and Asian typological structures.
  • Domain Adaptation: If specific corpora are available, MTQE can calculate a composite quality score to verify terminological adherence and domain consistency.
  • Scalable Architecture: Infrastructure designed for efficient handling of large text volumes allowing for incremental adaptation of its models.

Quantitative Scoring

  • Segment Level: Generates a normalized quality score in the [0, 100] range. This score correlates with the level of post-editing effort required.
  • Document Level: Calculates a consolidated quality metric for full files, facilitating accurate workload estimation and cost calculation.

Results and Operational Applications

  • Data Filtering and Curation: Facilitates the selection and filtering of high-quality parallel data for the finetuning of MT models.
  • Dynamic Engine Selection: Can implement engine switching to select the engine with optimal performance for a particular domain or language.
  • Post-editing Flow Optimization: Scores are used to classify content, allowing human editors to prioritize segments presenting greater difficulty.
  • Operational Risk Mitigation: Provides a predictive quality assessment before final delivery to prevent the dissemination of defective translations.

Technological Basis

Pangeanic's MTQE system undergoes a continuous training process on large datasets containing human-generated quality evaluations, including post-editing effort data and direct assessment scores. This ensures a high statistical correlation with human perception of translation quality.