EvaluateTranslation is a complex linguistic process. Its aim is to provide a comprehensible text in a foreign language. Quality is probably the most complex variable in this process. While specifying the cost or measuring the speed of translating a document is trivial, assessing the quality of a translated document is much more difficult. It is also an expensive process since automated metrics are not yet fully reliable.

TAUS supports buyers and providers of translation with the Dynamic Quality Framework: a comprehensive set of tools, best practices, metrics, reports and data to help the industry set benchmarking standards. The DQF Content Profiling wizard is used to help select the most appropriate quality evaluation model for specific requirements. This leads to the Knowledge base where you find best practices, metrics, step-by-step guides, reference templates, and use cases. Join the TAUS Evaluate Community and participate in our Translation Quality webinars!

USE CASES

The use cases illustrate examples of Quality Evaluation strategies implemented by members and non-members using TAUS resources.

CAPITA conducting a pilot with the DQF tools

Generic or Customized engines ie. how to provide valid benchmarking results when comparing MT output. Machine translation (MT) has been the hot-topic of the localization industry for some time now. Buyers of localization know they should be maximizing the use of MT in their workflows, but it can be difficult to decide how, when and where to use it. Building an MT infrastructure and deploying it as part of the translation workflow comes at a cost – and it can be very difficult to calculate ROI. Capita Translation and Interpreting (Capita TI) has conducted a pilot and showed in this webinar [[LINK]] how the TAUS DQF tools were used to evaluate the outputs of MT engines consistently. Results of Productivity and Quality tests in DQF helped one of Capita’s customer decide which engine to go with and what the associated ROI would be.

Ignacio Garcia using DQF in the translation classroom

In a TAUS webinar [[LINK]] Ignacio Garcia from University of Western Sydney explained how DQF was used in a translation technology class to conduct a pilot project. The aims were to provide feedback to the developers of DQF and to show how this type of tools can be integrated in the curriculum of translation studies. The project involved two evaluation tasks: productivity testing and MT comparison/evaluation based on MT output in Japanese, Chineese, Arabic and Spanish of small samples produced by Google translate and Bing translator. Students found the tools easy to use and a good starting point for collecting research materials on MT and Post-editing.

PayPal using DQF MT ranking

PayPal, one of the TAUS members that participates in the DQF workgroup, used several of the DQF tools to make a decision on MT technology to use. PayPal compared MT systems (in-house versus Commercial) using the DQF tools and automatic scores. They eliminated quick compare since it does not allow MT generated segments of two MT systems to be considered of the same quality.

Pactera conducting productivity test in DQF

Pactera, a large LSP and member of the DQF work group, wanted to establish productivity gains for post-editing. A second objective was to establish guidelines for pricing of post-editing both for translators and buyers. Pactera used the DQF productivity tools with a technical content sample and three European languages. They used two post-editors although they recognized that that is not enough to come up with a reasonable Inter Annotator Agreement: the gains varied widely between the two post-editors. Nevertheless, the results found by language pair are in agreement with those found by others. In addition to determining productivity gain and time gained, Pactera also calculated edit distance in order to find correlation between DQF results and the edit distance.

Systran partners with TAUS on DQF

Since December, 2013, Systran has been running a Progressive Testing Program designed for LSPs who wish to test drive the Hybrid MT technology at their own pace. To ensure customers make informed decisions, Systran decided to partner with TAUS in offering vendor-neutral tools and training for quality evaluation and post-editing. The TAUS resources and tools, in particular the Dynamic Quality Framework, provide efficient ways to quantify precisely the gains generated by SYSTRAN’s Hybrid technology.

VMware implements the DQF error-typology template

VMware successfully implemented an error typology derived from the DQF template. The TAUS content profiler helped creating a customized QE approach for both marketing and training content. VMware intends to continually refine their linguistic quality review processes and to follow closely any new developments in TAUS’s Dynamic Quality Framework, especially in the areas of systematic sampling and other review techniques such as readability and accuracy assessments.

Appen and CrossLang partner with TAUS on offering third party evaluation services

In October 2014, TAUS signed partnership agreements with the two linguistic consulting companies Appen and CrossLang who will provide translation quality evaluation services to TAUS members through the Dynamic Quality Framework (DQF). Until now members assigned the evaluation tasks to their own staff members or contractors. From now on users can choose to assign evaluation tasks to Appen and CrossLang from within the DQF platform. TAUS is pleased to complement DQF with the specialized linguistic consulting services. TAUS members do not always have the right resources internally to take care of quality evaluation or they prefer to outsource this vital task to independent specialists. 

TRASILT research team (Université Rennes 2) using DQF for comparing MT engines

Octavia Efraim from Université Rennes 2 conducted research towards the integration of MT into an automated online assistant application, with a view to rendering the monolingual agent bilingual. Part of the research was dynamic MT evaluation using the DQF tools with a view on fitness for purpose. The two main evaluation criteria were output quality and productivity. The research project has been successfully conducted using the DQF tools and have resulted in suggestions for improvements and new features for the DQF. 

LDS Church moves from own evaluation tool to DQF

For free, voluntary translation quality evaluation, LDS Church has been using an online evaluation platform developed internally. The TAUS Dynamic Quality Framework has been only used for paid evaluation but on a lower scale. Thanks to the comprehensive graphical reporting and the user-friendly interface of the DQF platform, LDS Church is now considering moving fully to DQF as it provides more flexibility and functionality than their in-house tool.

Read some testimonials on TAUS Evaluate and DQF here!