Dicta-Sign is expected to result in three proof-of-concept end user applications: a sign language-to-sign language terminology translator, a search-by-example tool, and a sign language Wiki. Aside from these applications, advances in the field are also expected to push major improvements to research and annotation tools.
Sign language-to-sign language terminology translator
Just like spoken languages, national signed languages differ from one another, and create communication barriers between deaf people of different nationalities. At the same time, European integration requires more and more communication across national boundaries.
Therefore, it is expected that a sign language-to-sign language translator will be useful in similar situations where web-based translators are useful. A prototype of such a tool will serve as the project demonstrator. It will be domain-specific, on the topic of travel, and make use of a controlled vocabulary of 1500 signs. The translation process itself will be shallow, based on simple relationships between the grammars of the four project sign languages, and on alignment of the parallel corpora. The input will be the results of the sign language recognizer, as well as annotated video, and the output will consist of avatar-based animations in the target language.
Search-by-example tool
Current lexical databases and dictionaries for sign language allow the user to search for signs by using another (written) language or by constraining some descriptive parameters. Both approaches require knowledge that cannot generally be assumed; especially in sign language learners who do not yet have a strong grasp of the parameters that make up a sign. A more promising alternative consists of having the user perform the sign in question in front of a camera.
A search-by-example system will integrate sign recognition for isolated signs with interfaces for searching an existing lexical database. Aside from the obvious utility to sign language learners, this prototype will also showcase the technology behind dictation characteristics of the user interface, where multiple alternatives are shown if a sign cannot be recognized reliably, as computationally this case is very similar to a learner performing a sign in a not-quite-correct manner.
Sign language Wiki
A major requirement of contemporary Web 2.0 applications is that user contributions are editable by an entire community. The oldest, and most popular, application of this type is a Wiki, where any contribution can be edited and refined, anonymously if so wished, by someone else. As the success of Wikipedia and related sites show, this type of community collaboration results in a rapid amassing of knowledge.
There is no doubt that sign language users could benefit similarly from collaborative editing. A server will be developed providing the same service as a traditional Wiki, but using sign language. Instead of using text as the output medium, a signing avatar presents information. The use of an avatar preserves the anonymity of the user, and facilitates modification and reuse of information present on the site.
The system acts as a dictation machine using sign, providing recording, playback, and editing. A user can put information onto the server using sign language by means of a web cam, or a video that is uploaded to the server. The system then analyzes the images and extracts key sign components, which are then used to generate the movements of the signing avatar. If the system interprets the sign language sentence badly, the user can correct it either by repeating only the erroneous part (for example the handshape, or facial expression).
This prototype will specifically showcase the integration of all major components of the project. At the same time, it will also demonstrate a Web 2.0 application that is accessible to the Deaf from the beginning to end.
