Thesaurus presents. Thesauri. linguistic principles of thesaurus construction. New explanatory and derivational dictionary of the Russian language, T. F. Efremova
Department of TAOY KemGUKI
Information retrieval thesauri:
structure, purpose and development procedure
1. Thesaurus as a way of systematized representation of knowledge and
kind of ideographic dictionary.
2. Information retrieval thesauri: essence and purpose
3. Structure of the IPT
4. The procedure for the development, examination, registration and maintenance of IPT.
Bibliography
1. GOST 7.74 - 96. Information retrieval languages. Terms and definitions [Text]. - Input. 1997-07-01. - Minsk: Interstate Council for Standardization, Metrology and, 1997. - 34 p. (System of standards for information, librarianship and publishing) TC 191.
2. GOST 7.25-2001. Thesaurus information retrieval monolingual. Development rules, structure, and presentation form [Text]. – GOST 7.25-80; Introduction 2002-07-01. - M.: IPK Publishing house of standards, 2001. - 16 p. MTK 191.
3. GOST 7.24-2007 Multilingual information retrieval thesaurus. Composition, structure and basic requirements for construction. - Instead of GOST 7.24-90; input. 2008-07-01. / Interstate Council for Standardization, Metrology and Certification. - M.: Standartinform, 2008. - 7 p. (System of standards on information, librarianship and publishing)
4. Baranov, O. S. Ideographic dictionary of the Russian language / O. S. Baranov. - M.: ETS Publishing House, 1995. - 820 s
5. Zhmailo, S. V. On the definition of the thesaurus [Text] / S. V. // NTI. Ser. 1 Organization and information work. - 2003. - No. 12. – P.20 – 25.
6. Zhmailo, S. V. Development of modern information retrieval thesauri [Text] / S. V. Zhmailo // NTI. Ser. 1 Organization and methodology of information work. -2004. - No. 1. – P.23 – 31.
So, in the ideographic dictionary of the Russian language by O. S. Baranov (4), 12 higher sections of the ideographic dictionary are distinguished, among which are: “order, nature, activity, culture”, etc., each of which is divided into groups, subgroups, departments, sections. All words in this dictionary are grouped into nests according to their meaning and are grouped by a certain concept with which they are most often associated by species relations. Nests are grouped into subsections and so on. At the moment, there are 5923 nests in the dictionary, 7 division levels (according to www.rifmovnik.ru/thesaurus.htm as of February 16, 2010). Here is an example of a dictionary entry from this dictionary:
178.4.7 aroma ▲ - a pleasant smell (for example, the smell of flowers, grass, hay. gentle #. intoxicating #). aromatization . . . ambre. incense.
The code of the word "aroma" reflects the ideographic classification accepted in this given word, in particular, the correlation of this word with the category "178-Sensations".
Thus, the terms "thesaurus", "ideographic dictionary", "thesaurus-type dictionary" primarily mean that the totality of the words of the language is presented in them in such a way that one group of words includes words that are similar in meaning. The main purpose of ideographic dictionaries is a collection of lexical units united by a common concept; this makes it easier for the reader to find the most appropriate means for adequate expression of thought and promotes active command of the language.
From the history of thesauri
|
JACKETS 2302 in Suits Coat products Sewing products n Double-breasted jacket Combined jacket Sports jacket in Packing measures Remaining material Waste material |
Lexical note; Ascriptors or descriptors-synonyms; Superior descriptors; Downstream descriptors; Associative descriptors; Descriptors linked by other kinds of relationships. Within each group of LUs associated with a head descriptor by one kind of paradigmatic relationship, there must be an alphabetical order of arrangement. For example: ALGORITHMIC LANGUAGES with algorithmic languages machine-oriented languages domain-specific languages in SOFTWARE FORMAL LANGUAGES n AUTOCODES a ALGORITHMS PROGRAMMING cf. artificial languages An ascriptor entry consists of an ascriptor and descriptors or a combination of descriptors that replace it when processing and searching for information. Here are examples of ascriptor articles: Alphanumeric characters Spanish FORMAL LANGUAGES NATURAL LANGUAGES see ALGORITHMIC LANGUAGES A dictionary entry may also include: How often the descriptor is used; Descriptor code number; Descriptor code according to the systematic index; Classification indices; Additional semantic and lexicographic marks; foreign equivalents. The quality of a lexico-semantic index is determined by the completeness of the lexical units included in it. is understood as the probability of entering into the thesaurus any informatively meaningful word for a given subject area. The completeness of the lexico-semantic index, and, consequently, of the entire thesaurus has a significant effect on the results of indexing documents and queries. Additional parts may include systematic, permutational, hierarchical and other indexes and lists of special categories of lexical units. A systematic index is an index in which descriptors are grouped according to the headings accepted in the IPT. A systematic index defines the thesaurus thematic direction, reveals its content and reflects those branches of science and technology that can be searched with one or another depth of detail. The need for it as part of the IPT is due to the fact that it gives a visual representation of the general state of terminology in a particular field of knowledge, allows you to build a coherent terminological model and, if possible, all the terms and concepts that should find a place in the thesaurus. It is intended to facilitate the search for terms when compiling search images of documents and queries by ordering a set of descriptors and ascriptors by subject. The systematic index, in essence, is a classification scheme for filling the thesaurus with terminology, since it is built by ordering a set of descriptors according to subject areas. Systematic indexes of IPT are divided into three types: Thematic, Mixed. This division reflects the principle of constructing the classification scheme of a systematic index. The main functions performed by the systematic index of IPT: Use as an auxiliary in indexing, providing, in total, the search for descriptors for indexing concepts that are not explicitly represented in the thesaurus (search function); Use in the process of maintaining a thesaurus (function of maintaining IPT); Use as a structural basis of IPT, as a management of its development (constructive function). In accordance with GOST 7.25-2001 (2), when constructing a systematic index of thematic and mixed types in its thematic part, rubrics of the Interstate NTI rubricator or a specific ASNTI rubricator compatible with the Interstate NTI rubricator should be used. When constructing a systematic index of categorical and mixed types, the following general categories follow in its categorical part: Names of disciplines and branches of activity; Items, materials; Methods, processes, operations, phenomena; Properties, values, parameters, characteristics; Relationships, structures, models, laws, rules, abstract concepts. Hierarchical index. A hierarchical index is an index that gives a list of lists of descriptors, each list starting with a descriptor that has no parent. It reflects the complete structure of hierarchical relationships in IPT. After each descriptor, descriptors are given directly with an indication of their level in the hierarchy by using numbering or a graphic designation of the level: The need to develop a hierarchical index of IPT is caused by the fact that the entire system of subordination of concepts is not fixed in the dictionary entries of IPT, because this would entail a significant increase in the lexico-semantic index. there is a need to develop an independent section of the IPT - a hierarchical index that would reflect the entire hierarchical chain of subordination of descriptors to the bottom. A permutation index is an index that lists in alphabetical order all the individual words that are part of the components of phrases denoting descriptors and for each of them all descriptors that include these words are indicated. Therefore, each term appears in the permutational index as many times as it contains significant words. The purpose of the permutational index is to provide a search for descriptors-phrases by any word included in their composition, including those that are not at the beginning of a lexical unit. It allows you to group single-root words in one place. As a rule, a permutation index is compiled in an automated way and usually has the form of an index of the KWIC type (Key Word - In Context - “Key words in context”), in which all significant words - terms - are arranged in alphabetical order. in the permutation index is in the center of the column, which is formed by the microcontexts of the term elements, and the part of the terms that does not fit is transferred to the left side of the same line:
|
Thesaurus(from Greek thesauros - treasure) in modern linguistics - a special kind of dictionaries of general or special vocabulary, which indicate semantic relationships (synonyms, antonyms, paronyms, hyponyms, hypernyms, etc.) between lexical units. Thus, thesauri, especially in electronic format, are one of the most effective tools for describing individual subject areas.
Unlike an explanatory dictionary, a thesaurus allows you to identify meaning not only with the help of a definition, but also by correlating a word with other concepts and their groups, which can be used in artificial intelligence systems.
In the past, the term thesaurus denoted primarily dictionaries that represented the vocabulary of the language with maximum completeness with examples of its use in texts.
Paronymy- partial sound similarity of words with their semantic difference (full or partial). Paronyms are often a source of speech errors.
Examples of single-root paronyms: dress - put on, human - humane, pay - pay - pay.
Examples of completely unrelated paronyms: biology - bryology, broth - brillon, compote - complot, texture - fracture.
However, a thesaurus is more than an information retrieval tool. Thesaurus can be considered as a universal model of a terminological system, and therefore - as a formal system of knowledge contained in the language of a particular scientific field.
General purpose thesaurus
Thesaurus in the most general definition is a dictionary with semantic links between vocabulary units. Since the late 1950s, thesauri have been used in machine translation systems and information retrieval systems (IPS).
Unlike semantic dictionaries, which are designed to describe general vocabulary in detail, thesauri are designed to store and classify extremely specific words and phrases. For example, the word substance is in the ROSS dictionary (Russian General Semantic Dictionary), and all the names of chemical compounds are already in the thesaurus.
What relationships are described in the thesaurus? Usually:
genus-species (AKO)
part-whole (POF)
synonymy/antonymy
associative.
An example of a genus-species relationship
Semantic parsing example
This paradigmatic(stable connections that exist between words in a language). And that's not all.
Syntagmatic(text) links are not represented in the thesaurus.
Example: WORDNET - intelligent computer thesaurus
http://wordnet.princeton.edu/perl/webwn
Created at Princeton University and distributed freely.
Key Features.
The words in it are grouped into synonymous groups ( synsets - synsets). They are divided into 4 dictionaries - nouns, adjectives, verbs and adverbs.
Synsets are united both in hierarchical relations (hyponyms and hypernyms), and in relation to antonymy and also meronymy (to be a part of something or to consist of parts).
The problem of morphology is also solved - the word after the call to WN returns in its original form.
Information retrieval thesaurus
In the field of information retrieval, thesauri benefit from the transition from text to descriptors that describe a real-world object. Jumping to descriptors allows for extended (redundant) indexing.
In the information retrieval thesaurus, PARADIGMATIC relationships between descriptors are explicitly expressed (not all, but those that are most often important for increasing the completeness of information retrieval). It has been experimentally determined that the most important paradigmatic relations are
subordination
resemblance
species-genus (genus-species)
cause-effect
part-whole.
Example of a dictionary entry:
Agreecultural machines. Agreecultural equipment
Syn. agricultural machinery, agricultural machinery,
View: potato harvester, seeder, etc.
An example of redundant indexing
Request "Agreecultural machines. Agreecultural equipment"
Example: Socio-Political Thesaurus of the Russian Language University Information System RUSSIA
http://www.cir.ru/index.jsp
Developed by the Autonomous non-profit organization "Center for Information Research" (ANO TsII)
Thesaurus is a terminological resource implemented as a dictionary of concepts and terms with links between them. The main purpose of the thesaurus is to help with information retrieval: based on the links of the thesaurus, the query is expanded, navigation through the links of the thesaurus helps to formulate the query itself more clearly.
A feature of the hierarchy of the Thesaurus UIS "Russia" is the plurality of classification, that is, for most concepts, not a single classifying concept is searched for (connection ABOVE - BELOW), but different points of view on a particular concept are described, for example, the concept of a SHOP can be considered both as a BUILDING and as a TRADING ORGANIZATION.
Thesaurus on socio-political topics, includes more than 26,000 concepts, 62,000 terms, 100,000 direct and 700,000 inherited relationships between concepts. The current version of the Thesaurus describes the terminology used in the socio-political field, including economic, political, military, legislative, social, international relations and other areas.
The full name of the Thesaurus is an information retrieval thesaurus on socio-political topics for automatic indexing. Here all definitions are important:
"information retrieval" - as it is designed specifically for use in information retrieval to help the user in the formation (clarification) of the request and to automatically expand the terms of the request during the search;
“on socio-political topics” - as it covers 95-99% of the vocabulary and terminology of the Russian-language text on socio-political topics;
”for automatic indexing” - as it is the basis for the process of automatically determining the subject of documents - grouping terms close in the thesaurus hierarchy into thematic nodes, automatic categorization and automatic annotation.
Thesauri - Conclusion
For many well-known thesauri (WordNet, Roget, EuroWordNet), automatic inference by thesaurus links remains a big problem - when the expansion to the nearest neighborhood is correct, but not complete, and attempts to expand the neighborhood lead to errors.
Conceptual system of the subject area The basis of any subject area is the system of concepts of this area. Definition of a concept: A concept is a thought that reflects objects and phenomena of reality in a generalized form by fixing their properties and relationships; the latter (properties and relations) appear in the concept as general and specific features correlated with classes of objects and phenomena (Linguistic Dictionary)
Concepts and terms To express the concept of a subject area in texts, words or phrases called terms are used. The set of terms of the subject area form its terminological system. The relationship of a specific term with other terms of the term system of the subject area is given by the definition
Definitions of the term? A word (or combination of words) that is an exact designation of a certain concept of any special field of science, technology, art, social life, etc. || A special word or expression used to denote something. in a particular environment, profession (Big Explanatory Dictionary of the Russian Language)
Terms - exact names of concepts Usually, each concept of the area corresponds to at least one unambiguously understood term, the meaning of which is this concept. - terms, in the sense of the traditional theory of terminology Properties of terms - the exact names of concepts - the term must refer directly to the concept, it must express the concept clearly; - the meaning of the term must be precise and must not overlap in meaning with other terms; - the meaning of the term should not depend on the context. Terms that accurately name a concept are the subject of study of the theory of terminology, terminologists
Text terms In real texts of the subject area, in addition to the main terms, a variety of different language expressions can be used to refer to a concept, which we call text terms: - syntactic and word-forming options: recipient of budget funds - budget recipient; - lexical variants - direct write-off, indisputable write-off; - multi-valued expressions, depending on the context, serving as a reference to different concepts of the region, for example, the word currency in different contexts can mean national currency or foreign currency.
Labeled descriptors Labels - part of the name of the descriptor cranes (lifting equipment) vs cranes (birds) shells (structures) - comparison of different thesauri Preferences for phrases: –Phonograph records vs. records (phonograph) Litters and plural: Wood (material) Woods (forested areas)
Inclusion of descriptors based on multi-word expressions Splitting the term increases the ambiguity: plant food The meaning of the expression depends on the word order: information science - scientific information One of the component words is outside the scope of the thesaurus or too general: first aid Descriptor relationships do not follow from its structure: –Artificial kidneys, refugee status, traffic lights
Associative relations Field of activity - character - Mathematics - mathematician Discipline - object of study - Neurology - nervous system Action - agent or tool - Hunting - hunter Action - result of action - Weaving - fabric Action - goal - Binding - book Cause-effect - Death - funeral Value - unit of measurement - Current strength - ampere Action - contractor - Allergen - anti-allergic drug, etc.
Information retrieval thesauri: stages of development Stage one: indexers describe the main topic of the text with arbitrary words and phrases Terms obtained from many texts are brought together Among terms that are close in meaning, the most representative is selected Some of the remaining ones become conditional synonyms, the rest are deleted Specific terms are usually not included
Information retrieval thesauri: the art of design Descriptors are terms that are needed to express the main topic of the document Synonyms are included only the most necessary (for example, start with a different letter) so as not to impede the work of the indexer Similar terms should be reduced to one term to avoid indexing subjectivity Hierarchy levels, inclusion of specific terms is limited
Information retrieval thesaurus: the art of development - 2 In complex cases, descriptors are supplied with labels and comments -LIV: bombardment - bombing -Ambiguous terms: one value in the thesaurus (capital), do not fit in the thesaurus, labels!!! Traditional information retrieval thesaurus - an artificial language built on the basis of real terms
Traditional IPT: application in automatic processing Lack of knowledge about the real language of the software Lack of knowledge of the real language of the software Legislative Indexing Vocabulary:Legislative Indexing Vocabulary: - in the text TROOPS - in the thesaurus MILITARY FORCES - in the text CAPITAL - the capital, in the thesaurus only the capital : polysemy or referring to different descriptors. But: polysemy or relating to different descriptors. Resolving ambiguity Resolving ambiguity
Traditional IPT: automatic query expansion Problem with associations Suggested: enter weights enter weights enter relationship names: object, property, etc. enter the names of relations: object, property, etc. CONCLUSION: you need to learn how to build linguistic resources specifically for automatic processing of text collections
Thesaurus EUROVOC – multilingual thesaurus of the European Community Thesaurus in 9 languages Russian version of EUROVOC –+5 thousand concepts reflecting Russian specifics Multilingual thesaurus –Descriptor – names in different languages –Ascriptors – for some languages
Rule-based automatic indexing on the EUROVOC thesaurus (Hlava, Heinebach, 1996) Rule example: IF (near "Technology" AND with "Development") USE Community program USE development aid ENDIF 40 thousand rules. Testing: the 20 most frequent descriptors in the text, generated automatically - 42% completeness, compared with manual rubrication
Automatic indexing based on establishing correspondence weights between words and descriptors (Steinberger et al., 2000) Stage 1 - establishing a correspondence between text words and assigned descriptors based on statistical measures (chi-square or log-likelihood) FISHERY MANAGEMENT descriptor - the following words (in descending order of weight): fishery, fish, stock, fishing, conservation, management, vessel, etc. 2nd stage indexing itself - summation of logarithms of weights or as a scalar product of vectors
Combination of loose and information retrieval thesaurus queries Manually indexed collection - correlations User sets natural language query Query is expanded by the thesaurus descriptors most strongly correlated with the query (Petras 2004; Petras 2005). For example, at the request of Insolvent Companies (Insolvent companies), a list of descriptors liquidity, indebtness, enterprise, firm. can be obtained, and the query is expanded. The accuracy in the experiment increased by 13%.
The section is very easy to use. In the proposed field, just enter the desired word, and we will give you a list of its meanings. I would like to note that our site provides data from various sources - encyclopedic, explanatory, word-building dictionaries. Here you can also get acquainted with examples of the use of the word you entered.
Meaning of the word thesaurus
thesaurus in the crossword dictionary
Explanatory dictionary of the Russian language. S.I. Ozhegov, N.Yu. Shvedova.
thesaurus
[te], -a, m. (special).
Dictionary of the language, which sets the task of a complete reflection of all its vocabulary.
A dictionary or set of data that fully covers the terms, concepts of some kind. special area.
adj. thesaurus, th, th.
New explanatory and derivational dictionary of the Russian language, T. F. Efremova.
thesaurus
Any dictionary. language, representing its vocabulary in full.
A complete, systematic set of data about a a field of knowledge that allows a person or a computer to navigate in it (in computer science).
Encyclopedic Dictionary, 1998
thesaurus
THESAURUS (from Greek thesauros - treasure)
a dictionary in which the words of the language are presented as fully as possible with examples of their use in the text (it is fully feasible only for dead languages).
A dictionary in which words related to any field of knowledge are arranged according to the thematic principle and semantic relations (genus-specific, synonymous, etc.) between lexical units are shown. In information retrieval thesauri, lexical units of text are replaced by descriptors.
Thesaurus
(from the Greek thesaurós ≈ treasure, treasury), a set of semantic units of a certain language with a system of semantic (see Semantics) relations given in it. T. actually determines the semantics of a language (a national language, the language of a specific science, or a formalized language for an automated control system). Initially, T. was considered as a monolingual dictionary, in which semantic relations are determined by the grouping of words according to thematic headings. For example, the English T. (author P. M. Roget), published in 1962 (1st edition 1852), contains 1040 headings, among which about 240,000 words are distributed. The index (key) to this T. contains an alphabetical list of words indicating the headings and subheadings to which each word belongs. There are traditional general language languages (descriptions of the semantic systems of individual languages) for English, French, and Spanish. Monolingual dictionaries that define the expressions of the main semantic parameters of each word are very close to T., for example, the dictionary of the Russian language by S. I. Ozhegov.
In the 70s. 20th century information retrieval volumes became widespread. Special lexical units, or descriptors, were identified in these volumes, which can be used to automatically search for documentary information. A synonymous descriptor is associated with each word of such a term (see Synonymy), and semantic relations are explicitly indicated for the descriptors: genus ≈ species, part ≈ whole, goal ≈ means, etc. It is usually customary to separate genus-species (hierarchical) and associative relations. Thus, the "Information Retrieval Thesaurus in Informatics", published in the USSR in 1973, provides for each descriptor a dictionary entry, which separately indicates synonymous keywords, generic, specific and associative descriptors. For better orientation in associative links between descriptors, semantic maps of thematic classes are attached to this T. In automated information retrieval, documents are searched for whose index contains not only query descriptors, but also those descriptors that are in certain semantic relationships with them. Sometimes it is useful to single out specific associative relations specific to a given thematic area in T.: disease ≈ causative agent, device ≈ purpose (or measured value), etc. The position of a lexical unit (word, phrase) in T. characterizes its meaning in the language; knowledge of the system of semantic relations into which a given word enters (including the rubrics where it enters) makes it possible to judge the meaning of this word.
In a broad sense, technology is interpreted as a description of the system of knowledge about reality that an individual carrier of information or a group of carriers possesses. This carrier can perform the functions of a receiver of additional information, as a result of which its T. also changes. The initial T. determines the capabilities of the receiver when it receives semantic information. In psychology and in the study of systems with artificial intelligence, the properties of the T. of individuals, which are manifested in the perception and understanding of information, are considered. In sociology and communication theory, they study the properties of T. of individuals and groups, which provide the possibility of mutual understanding based on the generality of T. In these situations, T. has to include complex statements and their semantic connections that determine the stock of information that a complex system has. T. actually contains not only information about reality, but also meta-information (information about information), which provides the possibility of receiving new messages.
Lit .: Cherny A.I., General methodology for constructing thesauri, “Scientific and technical information. Ser. 2", 1968, ╧5; Varga D., Methodology for preparing information thesauri, trans. [from Hung.], M., 1970; Shreider Yu. A., Thesauri in informatics and theoretical semantics, “Scientific and technical information. Ser. 2", 1971, ╧ Z.
Yu. A. Schreider.
Wikipedia
Thesaurus
Thesaurus, in the general sense - special terminology, more strictly and substantively - a dictionary, a collection of information, a corpus or code that fully covers the concepts, definitions and terms of a special field of knowledge or field of activity, which should contribute to correct lexical, corporate communication; in modern linguistics, a special kind of dictionaries that indicate semantic relationships (synonyms, antonyms, paronyms, hyponyms, hypernyms, etc.) between lexical units. Thesauri are one of the most effective tools for describing individual subject areas.
Unlike an explanatory dictionary, a thesaurus makes it possible to reveal the meaning not only with the help of a definition, but also by correlating a word with other concepts and their groups, due to which it can be used to fill the knowledge bases of artificial intelligence systems.
In the past, the term thesaurus dictionaries were designated mainly, representing the vocabulary of the language with examples of its use in texts with maximum completeness.
Also term thesaurus used in information theory to refer to the totality of all the information that the subject possesses.
In psychology, the thesaurus of an individual characterizes the perception and understanding of information. Communication theory also considers the general thesaurus of a complex system, through which its elements interact.
Thesaurus (disambiguation)
Thesaurus:
- Thesaurus - a dictionary, a collection of information covering the concepts, definitions and terms of a special field of knowledge or field of activity.
- Roger's thesaurus is one of the first and most famous ideographic dictionaries in history.
Examples of the use of the word thesaurus in the literature.
For perception and co-creation, some optimal thesaurus Not small, but not too big either.
With an unlimited amount of incoming information, significantly exceeding thesaurus, its value does not depend on this quantity and is entirely determined by thesaurus ohm.
The versatility, systemic nature of art leads to uneven perception of the work as a whole: for the perception of some aspects of the verse thesaurus optimal, for others, insufficient or too large.
Because thesaurus grows and changes, re-acquaintance with the work can mean receiving new valuable information.
The desire of the child to repeatedly reread the fairy tale he has grown fond of is understandable: his thesaurus his capacity for co-creation, for associative fantasizing is especially great.
This side of the matter is more changeable and subjective than thesaurus, and in search of an objective aesthetic evaluation of the work, it should be reduced to a minimum.
He penetrates into thesaurus poet and addresses the translation thesaurus from a foreign reader.
This most important thing is to determine how big your thesaurus, T.
No, it's just that his own baggage is scanty, he is undeveloped, his thesaurus is in its infancy, and if he does not understand that thesaurus should be increased, then, in any case, this woman will have a hard time with him.
Rich thesaurus, based on true knowledge, allows a person in communication with another person, including in the closest communication with the closest person, to respond correctly to everything that happens.
Obviously, the fall in the value of information with the growth thesaurus should depend on the relationship thesaurus to the amount of information received.
Obviously, the optimal value of artistic information corresponds to the proximity thesaurus reader and thesaurus poet.
We can say that co-creation, like creativity, requires inspiration, that is, the inclusion thesaurus in the broadest sense of the word.
Such an internal repetition of bright imagery and bright sound, remaining within the existing thesaurus, enriches it with the same aesthetic moment of repetition.
At this point thesaurus Nabokov and Prishvin should be considered antipodes of Platonov, and Marina Tsvetaeva can be recognized as similar to him.
N. V. Lukashevich
B. V. Dobrov
Research Computing Center of Moscow State University M.V. Lomonosov;
ANO Center for Information Research
Keywords: thesaurus, information retrieval, automatic text processing,
The vast majority of technologies that work with large collections of texts are based on statistical and probabilistic methods. This is due to the fact that lexical resources that could be used to process text collections using linguistic methods should have a volume of tens of thousands of dictionary entries and have a number of important properties that need to be specially monitored when developing a resource. In the report, we consider the basic principles of developing lexical resources for automatic processing of large text collections using the example of the Russian language thesaurus created since 1997 for computer processing of texts RuThez, which is currently a hierarchical network of more than 42 thousand concepts. We describe the current state of the thesaurus based on a comparison of its lexical composition and the text corpus of the University Information System RUSSIA (www.cir.ru) - 400 thousand documents. Examples of using the thesaurus in various automatic word processing applications are discussed.
- Introduction
Currently, millions of documents have become available in electronic form, thousands of information systems and electronic libraries have been created. At the same time, information systems that use lexical and terminological resources for searching are calculated in fractions of a percent. This is due to the serious problems of creating such linguistic resources for the automatic processing of modern collections of electronic documents.
First, these collections are usually very large, the resource must include descriptions of thousands of words and terms. Secondly, collections are a set of documents of different structure with a variety of syntactic constructions, which makes it difficult to automatically process text sentences. In addition, important information is often distributed among different sentences of the text.
All this sharply raises the question of what kind of linguistic resource should be, which, on the one hand, would be useful for automatic processing and searching in electronic collections, on the other hand, could be created in a foreseeable time and maintained with relatively little effort.
In the article, we will consider the basic principles of developing lexical resources for automatic processing of large text collections. These principles will be considered on the example of the thesaurus of the Russian language created since 1997 by the ANO Center for Information Research for computer processing of texts RuThez. RuThez is currently a hierarchical network of more than 42 thousand concepts, which includes more than 95 thousand Russian words, expressions, terms. We will describe the current state of the thesaurus based on a comparison of its lexical composition and the lexicon of the text corpus of the University Information System RUSSIA, supported by the Research and Development Center of Moscow State University. M.V. Lomonosov and ANO TsII. UIS RUSSIA (www.cir.ru) contains 400,000 documents on socio-political topics (about 3 GB of texts, 200 million word usages). The article will also look at examples of using the thesaurus in various word processing applications.
- Principles for the development of a linguistic resource
for information retrieval tasks
To ensure efficient automatic processing of electronic documents (automatic indexing, categorization, comparison of documents), it is necessary to build a basis for their comparison - a list of what was mentioned in the document. For such an index to be more effective than a word index, it is necessary to overcome the lexical diversity of the text: synonyms, polysemy, parts of speech, style, and reduce it to an invariant - a concept that becomes the basis for comparing different texts. Thus, concepts should become the basis of a linguistic resource, and language expressions: words, terms - become only text inputs that initialize the corresponding concept.
To be able to compare different, but close in meaning, concepts, relationships must be established between them. Traditionally, in linguistic resources for automatic processing of texts in natural language, certain sets of semantic relations were used, such as part, source, cause and so on. However, when working with large and heterogeneous text collections, we must understand that with the current state of text processing technology, a computer system will not be able to detect these relationships in the text in any stable way in order to perform the procedures that we have associated with certain relationships. Therefore, relations between concepts should first of all describe some invariant properties that do not depend or weakly depend on the topic of a particular text in which the concept is mentioned.
The main function of these relations is to answer the following question:
if it is known that the text is devoted to the discussion of C1, and C2 is connected
attitudeRwith C1, can we say that the subject of the text(*)
has to do with C2?
When creating a linguistic resource for automatic processing, it is important to determine which properties of the concepts C1 and C2 allow establishing the correct (*) relations between them.
So, for example, whatever texts are written about birches, we can always say that these are lyrics about trees. But despite the popularity and frequent discussion of the relationship tree as part forests, a very small number of texts about trees are texts about forests. Note that the problem is not related to the name of the relationship. So clearing is part of the forest, and texts about clearings are texts about the forest.
The invariance of relations with respect to the spectrum of possible topics of the texts of the subject area is largely determined by deeper properties than those reflected by the names of the relations, namely its quantifier and existential properties. So the quantifier properties of relations describe whether all instances of a concept have a given relation, whether a given relation is preserved throughout the entire life cycle of the example. Problem using relation tree – forest it is precisely connected with the fact that not every particular tree is in the forest, but the clearing cannot be outside the forest.
An example of describing the existential properties of relations is whether the existence of the concept C2 follows from the existence of the concept C1 (for example, the existence of the concept GARAGE requires the concept AUTOMOBILE) or the existence of examples of C1 depends on the existence of examples of C2 (so a particular FLOOD inseparable from a concrete example RIVERS). The discussion in the text of the dependent concept C2, especially the example dependent one, suggests that the text is also relevant to the main concept C1.
Consider the relationship between the concepts FOREST and WOOD in details. In fact, part of the concept FOREST is TREE IN THE FOREST, while there are and STANDING TREE,TREE IN THE GARDEN etc. In any case, it is required to break the relation of subordination of the concept TREE notion FOREST.
On the other side, FOREST is kind SET OF TREES, does not exist without trees (as well as GARDEN). Thus the concept FOREST should be dependent on the concept TREE. Starting with an analysis of the needs of specific applied tasks, we came to the conclusion that it is important to describe the deep properties of relationships that were previously very insignificantly reflected in linguistic resources, but which are of paramount importance for tasks of automatic processing of large text collections, and, possibly, for many other tasks.
Now we are modeling the description of quantifier and existential properties of concepts by a set of traditional thesaurus relations ABOVE-BELOW (66% of all connections), PART-Whole (30% of connections), ASSOCIATION (4%), in combination with some set of additional modifiers (20% of relations are labeled). Note that the PART-Whole and ASSOCIATION relations are interpreted according to the rule (*). In total, about 160 thousand direct connections between concepts are described, which, taking into account the transitivity of relations, gives a total number of different connections of more than 1350 thousand connections, that is, on average, each concept is connected with 30 others.
- RuThes Thesaurus: General Structure
The RuThes Thesaurus is a hierarchical network of concepts corresponding to the meanings of individual words, textual expressions or synonymous series. Thus, the main elements of the thesaurus are concepts, language expressions, relations, language expression - concept, relationships between concepts.
In the thesaurus, both linguistic knowledge - descriptions of lexemes, idioms and their connections, traditionally related to lexical, semantic knowledge, and knowledge about terms and relationships within subject areas, traditionally related to the field of activity of terminologists, described in information retrieval thesauri, are collected in a single system. As such subject subdomains, the thesaurus describes such subject areas as economics, legislation, finance, international relations, which are so important for a person's daily life that they have a significant lexical representation in traditional explanatory dictionaries. In them, lexical and terminological are strongly interconnected and strongly interact with each other.
Language expressions are separate lexemes (nouns, adjectives and verbs), nominal and verbal groups. Thus, the thesaurus now does not include adverbs and auxiliary words as linguistic expressions. Multi-word groups may include terms, idioms, lexical functions ( influence e).
For each language expression, the following is described:
Its ambiguity is the connection with one or more concepts, which means that a given linguistic expression can serve as a textual expression of this concept. The assignment of a linguistic expression to different concepts is also an implicit indication of its ambiguity;
Its morphological composition (part of speech, number, case);
Features of writing (for example, with a capital letter), etc.
Each thesaurus concept has a unique name, a list of language expressions by which this concept can be expressed in the text, a list of relationships with other concepts.
As a unique name for a concept, one of its unambiguous textual expressions is usually chosen. But the name of the concept can also be formed by a pair of its ambiguous textual expressions - synonyms written with a comma and uniquely defining it (for example, the concept FAT, FAT). An ambiguous textual expression of the name of a concept can also be provided with a label or a shortened fragment of interpretation, for example, the concept CROWD (CLUSTER OF PEOPLE).
- Example of a dictionary entry
We have chosen as an example the dictionary entry of the concept FOREST corresponding to one of the meanings of the word forest. This dictionary entry is interesting because it includes different types of knowledge traditionally referred to as lexical (semantic) knowledge and encyclopedic knowledge (knowledge about the subject area, terminology).
Synonyms for the concept FOREST(total 13):
forest(M), forest zone, forest environment,
forest, forest quarter, forest landscape,
forest area, forest, forested,
forest raw area, forest,
array of forests.
The following terms with synonyms:
JUNGLE(jungle);
FOREST PARK(city garden, green area,
green massif, forest park,
forestry, forestry
belt, parkM), park zone);
FOREST HUNTING;
deciduous forest(softwood forest, hardwood
forest);
GROVE(oak forest);
CONIFEROUS FOREST (coniferous massif, dark coniferous forest)
Concepts-parts with synonyms:
BORELOM (windbreak, windfall);
FELLING(cutting area);
FOREST CULTURE(forest species, forestry
culture);
FOREST LAND (lands of the forest fund; lands covered with
forest; forest land, forest area;
wooded land, wooded
area,);
FOREST(forest plantations, forest plantations,
afforestation);
FOREST EDGE(edging, edging);
UNDERGROWTH (undergrowth);
PROSECA;
DRY LAND(dry).
Here the symbols (M) reflect the mark of the ambiguity of the text input.
concept FOREST also has other relationships, the so-called dependency relationships (in the modern version they are called ASC 2 - asymmetric association): FOREST FIRE(forest fire, fire in the forest; FOREST MANAGEMENT (forest use, use of forest fund plots); FOREST OWNERSHIP; FOREST SCIENCE (forest science). As already noted in paragraph 2, the concept of FOREST depends on the concept of TREE, which in the thesaurus is denoted by the relation ASC 1 .
Whole concept FOREST is directly related to 28 other concepts, taking into account the transitivity of relations - with 235 concepts (more than 650 text inputs in total).
- Assessment of the state of the art
Thesaurus of the Russian language RuThez
5.1. Lexical composition
Currently, more than 95 thousand language expressions are included in the thesaurus network, of which 61 thousand are single-word ones.
This amount of work made us decide what words and language expressions should be included in the descriptions of the Thesaurus. The natural desire was to see how the most frequent words of the Russian language are represented in the thesaurus. For this, the text collection of the University Information System RUSSIA (400 thousand documents) was used. The collection contains official documents of various bodies of the Russian Federation (55,000 documents since 1992), as well as press materials since 1999 (newspapers Izvestia, Nezavisimaya Gazeta, Komsomolskaya Pravda, Arguments and Facts, Expert magazine, and others), materials from scientific journals (Bulletin of the Moscow University, Sociological Journal). The comparison was made between the list of lemmas included in the Thesaurus and the list of the most frequent 100,000 lemmas in the text collection (frequency more than 25).
The lexical markup of the list showed that among these one hundred thousand lemmas, 35 thousand are described in RuThes, only about 7 thousand lexemes deserve to be included in the Thesaurus, the rest are lemmatic variants of various proper names. Therefore, replenishment has ceased to be a priority and is carried out gradually, starting with the most frequent words. It is assumed that as soon as this list is basically exhausted, the next comparison with the text array of the information system will be performed, new tokens with a frequency of more than 25 will be selected. Further, the viewing threshold is supposed to be reduced. The presence in the text collection of a large number of text examples allows you to quickly respond to "lexical novelties" (for example, installation,blockbuster, beau monde, thriller) and include them in the appropriate places in the hierarchical system of the Thesaurus.
Constant work with the current text collection provides unique opportunities to test the significance and quality of lexical descriptions offered in dictionaries. For example, an unusually high frequency of use of the word Mother See(more than 400 times). Checking against the array showed that the word is indeed often used as a synonym for the word Moscow, while explanatory dictionaries often mark this word as obsolete. Another example of a frequently used word (more than 300 times) marked as obsolete in dictionaries is the word blissful.
5.2 Description of word meanings
A comparison with the text collection shows that many of the frequency words in the array are well represented in the Thesaurus in at least one of their (usually basic) values. Finding out to what extent the range of meanings of polysemantic words of the Russian language is represented in the Thesaurus is our primary task at the present time.
As you know, different dictionary sources often give a different set of meanings for polysemantic words, distinguish shades of meanings, and the same type of polysemy can be described differently for different words even in the same dictionary. Therefore, the task of a consistent and representative description of the meanings of lexemes is an important task for the creators of any dictionary resource.
However, if the resource is intended for automatic processing, then the task of balanced description of values becomes much more important. Excessive inflating of values can cause the computer system to be unable to select the desired value, which in turn leads to a significant decrease in the efficiency of the automatic word processing system. So, as one of the disadvantages of the WordNet resource as a resource for automatic word processing is an excessive number of values described for some words (in WordNet 1.6: 53 values for run.47 for play and so on.). These meanings are difficult to distinguish even for a person when semantic annotating texts. It is clear that the computer system also cannot cope with the choice of an appropriate value. Therefore, different authors propose different ways of combining values to improve the quality of processing.
At the same time, the opposite factor acts: if the values really differ in their set of vocabulary links (in our case, thesaurus links) - they cannot be glued into one unit (one concept) - this will also lead to a deterioration in the quality of automatic processing.
Consider for example the words school And church, each of which can be considered as an organization and as a building.
Each school organization has a building (most often one). All parts of the school building (classrooms, blackboards) are related to school as an organization. There are no specific types of school buildings. Therefore the description schools as buildings it is inappropriate to single out as a separate concept. However, the description of such a cumulative concept SCHOOL as an organization and as a building must have a specially designed relationship with the concept BUILDING. When describing such relationships in the Thesaurus, a mark on the relationship is used - the modifier “A” (“aspect”, in automatic analysis, to take into account this relationship, “confirmation” by other concepts is required).
SCHOOL
HIGHER EDUCATIONAL INSTITUTION
ABOVE A PUBLIC BUILDING
Relevant word meanings church not so close. churches How an organization can have a large number of church-buildings in different locations and also have many other buildings. church-building closely associated with religion and confession, but can change belonging to organization churches. church-organization And church-building have different subspecies. That's why CHURCH (ORGANIZATION) And CHURCH (BUILDING) are presented in RuThes as different concepts.
The significant divergence in thesaurus relationships correlates in an interesting way with the ability of denotations corresponding to meanings to exist separately from each other. Thus, the church-building does not cease to exist and even be called a church even when the use changes, unlike the school-building.
The process of reconciliation of the representation of values in the Thesaurus is constantly being carried out, starting with the most frequent lemmas. For each frequency token, it is checked how its values are described in explanatory dictionaries, what values are used in the collection and how they are presented in the Thesaurus. As a result, a list of 10,000 lexemes has been formed, the ambiguity of which still requires either additional analysis or additional description. The list is based on 30 thousand of the most frequent lemmas.
It should be noted that in the Thesaurus the problem of ambiguity is partially removed due to the fact that thesaurus relationships can be described between the different meanings of a word, and therefore the highest concept in the hierarchy can be chosen by default. It was definitely discussed in the text. For example, the word photo has three meanings: photography as a field of activity, photography as a photograph, photography as a photo studio:
PHOTOGRAPHY(photographing, photography, ..., photo )
PART PHOTOGRAPHIC IMAGE
(photo, photograph, photo )
PART PHOTO STUDIO (photo ).
Thus, if it was not possible to figure out what meaning the word is used photo, the default is considered to be a photograph (process, result, or location), which is sufficient for many automatic word processing applications.
- Application of the RuThes thesaurus
for automatic word processing
Since 1995, RuThes socio-political terminology (socio-political thesaurus) has been actively and successfully used for various applications of automatic text processing, such as automatic conceptual indexing, automatic categorization using several rubricators, automatic annotation of texts, including English ones.. Socio-political thesaurus (27,000 concepts, 62,000 text entries) is the basic search tool in the UIS RUSSIA search system (www.cir.ru).
The entire vocabulary of the RuThes thesaurus is used in the procedures for automatic rubrication of texts according to complex hierarchical headings. In the existing technology, each rubric is described as a Boolean expression of terms, after which the original formula is expanded along the thesaurus hierarchy. The resulting Boolean expression may already include hundreds and thousands of conjuncts and clauses.
Let us give as an example a fragment of the description by thesaurus concepts (and language expressions after the expansion of the formula) of the “Image of a Woman” rubric of the SOFIST 2 rubricator used by VTsIOM to classify public opinion survey questionnaires:
(WOMAN[N]
|| GIRL[N]
|| RELATIVE[L] (grandmother, granddaughter, cousin,
daughter, sister-in-law, mother, stepmother, daughter-in-law, stepdaughter, ...))
(CHARACTER TRAIT[L] (thrifty, heartless, forgetful,
frivolous, mocking, intolerant, sociable, ...)
|| IMAGE[E] (representation, appearance, appearance,
appearance, shape, image, appearance)
|| PLEASANT[L] (..., interesting, beautiful, cute,
attractive, attractive, endearing, ...)
|| UNPLEASANT[L] (unsympathetic, rude, nasty, ...)
|| VALUE [L] (revere, idolize, adore,
worship, worship, ...)
|| PREFER[N]
The symbol "E" denotes the full expansion along the thesaurus hierarchy, the symbol "L" - according to species relationships ("BELOW"), the symbol "N" - do not expand.
Research is being carried out on the development of a combined technology for automatic text categorization that combines thesaurus knowledge and machine learning procedures.
The issues of using a thesaurus to expand a query formulated in natural language (now only the socio-political part of the thesaurus is used to expand the terminological query in the information retrieval system of the UIS RUSSIA), searching for answers to questions in large text collections.
7. Conclusion
The paper presents the basic principles of developing linguistic resources for automatic processing of large text collections. The created linguistic resource - RuThes Russian Thesaurus - is intended for use in such applications of automatic text processing as conceptual indexing of documents, automatic rubrication by complex hierarchical headings, automatic expansion of natural language queries.
This work is partially supported by the Russian Foundation for the Humanities, grant No. 00-04-00272a.
Literature
- Lukashevich N.V., Saliy A.D., Knowledge representation in automatic text processing //NTI, Ser.2. 1997. No. 3. S. 1-6.
- Zhuravlev S.V., Yudina T.N., Information system RUSSIA //NTI, Ser.2. 1995. No. 3. S. 18‑20.
- Winston M., Chaffin R., Herman D., A Taxonomy of Part-Whole Relations // Cognitive Science. 1987. no. 11. P. 417-444.
- Priss U.E., The Formalization of WordNet by Methods of Relational Concept Analysis // WordNet. An Electronic Lexical Database / Ed. by C. Fellbaum. Cambridge, Massachusetts, London, England.: The MIT Press 1998. P. 179-196.
- Guarino N., Welty C., A Formal Ontology of Properties // Proceedings of the ECAI-00 Workshop on Applications of Ontologies and Problem Solving Methods. Berlin: 2000. P. 121-128. (http://citeseer.nj.nec.com/guarino00formal.html).
Some Ontological Principles for Designing Upper Level Lexical Resources // First Int. Conf. on Language Resources and Evaluation. 1998.
- LukashevichN.V., Dobrov B.V., Modifiers of conceptual relations in the thesaurus for automatic indexing // NTI, Ser.2. 2000, No. 4, S. 21-28.
- Big Explanatory Dictionary of the Russian Language / Ed. S.A. Kuznetsova. St. Petersburg: Norint, 1998.
- Ozhegov S.I., Shvedova N.Yu., Explanatory dictionary of the Russian language - 3rd edition. M.: Az, 1996.
- Apresyan Yu.D., Selected works, volume I. Lexical semantics: 2nd ed. M.: School "Languages of Russian culture", Ed. Firm "Eastern Literature" RAS, 1995.
- G. Miller, R. Beckwith, C. Fellbaum, D. Gross and K. Miller, Five papers on WordNet, CSL Report 43. Cognitive Science Laboratory, Princeton University, 1990.
- Chugur, J. Gonzalo and F. Verdjeo, Sense distinctions in NLP applications // Proceedings of “OntoLex-2000”: Ontologies and Lexical Knowledge Bases. Sofia: OntoTextLab. 2000.
- Loukachevitch N., Dobrov B., Thesaurus-Based Structural Thematic Summary in Multilingual Information Systems // Machine Translation Review. 2000 No. 11. P. 10-20. (http://www.bcs.org.uk/siggroup/nalatran/mtreview/mtr-11/mtr-11-8.htm).
Thesaurus of russian language for natural language processing
of large text collections
Natalia V. Loukachevitch, Boris V. Dobrov
keywords: thesaurus, natural language processing, informational retrieval
In our presentation we consider main principles of developing lexical resources for automatic processing of large text collections and describe the structure of Thesaurus of Russian Language, which is developed since 1997 specially as a tool for automatic text processing. Now the Thesaurus is a hierarchical net of 42 thousand concepts. We describe the current stage of the Thesaurus developing in comparison with 100,000 the most frequent lemmas of the text collection of University Information System RUSSIA (www.cir.ru), including 400 thousand documents. Also we consider the use of the Thesaurus in different applications of automatic text processing.
