Natural Language Corpora Guidance

Field	Purpose of Use
Language Research	General linguistics、Japanese Linguistics、Language study, and other areas of natural language research.Comparative linguistics via the comparison of multiple language corpora.
Information Processing	Language modeling for speech recognition、creation of acoustic models.Construction of models for natural language processing.Speech synthesis based on manipulation of phonemic units.
Language Education	Development of materials for teaching Japanese as a foreign language.Support for native Japanese language learning.
Language Policy	Basic data for studying the table of kanji for common usage.Review data regarding notation in official (government) statements.
Lexography	Example lookup. Clarification of relationships between words.
Psychology	Design of experiments based on language, stimulus control.

Creating a Corpus

There are various methods of creating a corpus depending upon the objectives and planned uses of said corpus, but this example will cover the methods used in creating the "Balanced Corpus of Contemporary Written Japanese".

1. Sampling Methods

Samples to be used in the corpus are chosen at random.

For example, in the case of samples from books, roughly 30,000 samples were chosen from books published from 1986 to 2005.

When samples are taken, a random character from a certain page is chosen as the sampling point.

In the corpus, there are two types of samples based on the sampling point.

2. Sample Creation

１）Decide on the section to be transcribed in the corpus

Once the sampling point has been chosen, the sample that will actually be used in the corpus is chosen.

２）Complete the copyright process

The copyright process must be completed for each sample. Samples without a license will not be accepted.

３. Digitizing Samples

Samples are then digitized in XML format.

４. Releasing the Data

Each corpus that is constructed will be sequentially made public in the following fashion. However, not all corpora will be made available as described below.

• Free Online Release (Some cases may require a usage contract)
• Paid Online Release (Usage contract required)
• Paid Release of All Data (DVD edition)

The free online release version "Shonagon" can be tried for free.

Corpus Release and Copyright

There is much merit in making corpora widely available. In order to accomplish this when a corpus contains data relating to modern published works which are still under copyright, it is necessary to obtain licenses from the original creator in order for the corpus to be used freely.

In the case of the "Balanced Corpus of Contemporary Written Japanese", over 100 million words of data have been taken from sources such as books, newspapers, and the internet where copyright has not lapsed. Therefore in order for the corpus we have constructed to be used as widely as possible, we have completed the copyright process for all data transcribed in the corpus.

Additional Information for Data Searches (Annotation)

In its simplest form, a corpus of written language is simply text (where samples are arranged as character strings), and can be thought of as a "Text Corpus". Such a corpus can be easily searched for simple and complex conditions using regular expressions.

However, in a simple text corpus, the word "分かち書き" ("word-separation"), for example, could be written in a number of ways in Japanese, such as: "分かち書き", "分ち書き", "わかち書き", "分かちがき, "分ちがき", "わかちがき", or "別ち書き", and in such a case it would be impossible to accurately search a text corpus using only the query "分かち書き". Because of this, there is a need to add further information to the corpus to. This additional mark-up work is called annotation.

The basic annotation process for Japanese "morphological (part of speech annotation) analysis", is as follows.

• Separate the text into "Words".
• Determine the part-of-speech of each word.
• Know the possible variations in representation for words, to decide on the identity of the word.

Aside from morphological analysis, there annotation regarding dependency structure, and predicate argument structure (i.e. subject-object-predicate), and for spoken-language corpora there can also be annotation of speech-to-text transcriptions, phonemes, and intonational information.

When constructing a large-scale corpus, developing an automated annotation system is necessary. For the purposes of morphological analysis, NINJAL and Chiba University have jointly developed technology for the UNIDIC morphological analysis dictionary and a morphological analyzer and have reached a level where practical application is possible.

The Future of Japanese Corpora

In the future, the use of corpora in language research will allow us to lend objective information to the issues of language policy and education, which have thus far been dependent on the opinions of so-called experts.

In terms of basic language research, research that stresses actual patterns of usage over only language structure, research that focuses on the quantitative side of language, and research that does not only look at the underlying systems of language but also at the relationship between language's social context will be facilitated.

In the future, as corpora become more connected to the internet, they will likely expand to billions, or tens of billions of total words. Because it would be virtually impossible for researchers to examine the entirety of the data in such a case, it will become necessary to automatically classify the data using natural language processing and statistical data analysis techniques. As a result, corpora will hopefully contribute to the lessening of barriers between humanities-focused and science or engineering-focused language research.