Lexical Data Software Development

Oxford Languages division of Oxford University Press (OUP) is the world leader in human language technology and the primary provider of quality lexical data to academics, technologists and businesses around the globe.

Within its Oxford Global Languages (OGL) initiative, Oxford Languages seeks to create a major digital repository of language data in hundreds of languages and make it available for researchers, learners and developers worldwide. The project started in 2015 and hasn’t slowed down since then. The results of this tremendous work — clean and structured language datasets — are used by engineers and innovators for different use cases, from lexical software development to powering machine translation, NLP and Machine Learning technologies.

Originally, Oxford Languages outsourced the conversion of each dictionary to different teams across the globe depending on the source language. Every team built its own parsing tool to process language data. These tools were incompatible and couldn’t be integrated into a unified workflow. This approach posed a set of challenges in terms of scalability, efficiency and quality of the conversion:

• Lack of automation and standardization. The conversion process was largely manual. It involved unique specialists and required custom lexical software development for each language and type of dictionary.
• Specifics of formats and data. Original data can be structured (eg. XML), unstructured (eg. PDF) and semistructured (eg. XML that requires parsing) and has to be transformed from different source formats to different target formats.
• Slow process, high cost. It took from 3 weeks to 3 months to create a well-structured and verified dataset for a digital dictionary.
• High error rate. Mostly manual process resulted in up to 20% data loss and a high error rate.

Oxford Languages dictionaries are licensed and used by startups and world-known tech giants such as Amazon, Apple, Google and Microsoft for different purposes, including the development of Natural Language Processing software, search engines, multilingual applications and machine translators. The workflow can be tailored depending on the requirements and goals of each conversion project and adjusted to address the needs of each customer precisely.

For example, the custom XML target format was added for Amazon projects. The workflow was repurposed to create wordlists using the Neo4j database to provide the other customers with quality language content.

Dictionaries Conversion Framework is the blend of data processing technology and modern engineering and QA practices. Digiteum team has evolved the framework and methodology along the 8-year journey, significantly improved conversion speed and performance. Only in 2020, the team has already produced about 90 dictionaries and counting, compared to the total 28 dictionaries issued in 2017.