ARCHER’s second evaluation campaign starts today, September 21st. Participants will have until October 2nd to submit their hypotheses. This campaign features four evaluation tasks: automatic speech recognition (ASR), machine translation (MT), named entity recognition (NER), and optical character recognition (OCR). The evaluation focuses on resources in French, English, German, and Russian, providing participants with an opportunity to assess their systems across a multilingual setting.
The ASR task consists of transcribing human speech into written text through a combination of speaker diarization and speech transcription. Each speech segment must be accurately transcribed and assigned a speaker label, enabling systems to distinguish between different speakers within a conversation.
The MT task aims to translate human-written text from one language into another. In this campaign, participants are required to translate documents across several language pairs: French to English, English to German, Russian to English, and German to French. These translation directions have been selected to evaluate the performance and robustness of machine translation systems in a multilingual environment.
The NER task focuses on the automatic identification and classification of named entities, that is, text segments referring to specific real-world or fictional entities. Examples include persons, locations, organizations, dates, and numerical expressions.
The OCR task involves converting typed, handwritten, or printed text images into machine-readable text. Participants will process images scanned from single-page and multi-page documents and generate text transcriptions that preserve the original reading order of each page.
Following the conclusion of the automatic evaluation campaign, a dry-run campaign dedicated to Human-in-the-Loop evaluation will take place from October 5th to October 15th. This exercise aims to establish and validate a first version of the evaluation protocol for measuring how effectively ASR and MT systems can collaborate with human users over time.
Human-in-the-Loop evaluation goes beyond measuring the quality of a system’s output in isolation. Instead, it assesses how a human analyst and an AI system work together, with the analyst providing corrections and feedback that can help improve the system’s future performance.
The use case being explored in ARCHER involves an analyst who processes a sequence of documents, for example one document per day. For each document, the analyst spends a limited amount of effort reviewing and correcting the system’s output. Once the work on a document is completed, the analyst does not revisit it. The objective is to evaluate whether the system can effectively learn from these interactions and make better use of the analyst’s corrections.
For the ASR task, this may involve adapting to recurring speakers, pronunciation patterns, or specialized terminology encountered by the analyst. For the MT task, systems may benefit from previously corrected translations, preferred terminology, and domain-specific language used across related documents.
The dry-run campaign will allow participants and organizers to test the evaluation methodology, identify potential challenges, and refine the protocol before future official Human-in-the-Loop evaluations. Ultimately, this work aims to better reflect real operational scenarios, where language technologies are not used as standalone tools but as collaborative assistants that continuously improve through interaction with human experts.
