Attention driven memory

Grunewalder, S. and Obermayer, K. (2005) Attention driven memory. In: CogSci 2005. Cognitive Science Society, pp. 845-850. ISBN 0976831813

Abstract

Categorization is a skill which is used extensively in everyday life and as therefore an important aspect of human cognition. Consequently a variety of studies exist which address the topic and revealed that diverse factors affect human categorization performance. A critical but not extensively studied factor is time. Imagine watching a basketball game for 30 minutes. In this period of time plenty of actions will take place resulting in diverse impressions which make you afterwards categorize the game as interesting or boring. Such a categorization task is very similar to a time series classification task in the context of machine learning. In the field of machine learning a phenomen called “vanishing gradient” is known which makes it generally hard to solve such a categorization task. A prominent method that overcomes this phenomen is the long short term memory which basicly consists of a memory that is controlled by two gate units which can be interpreted as adaptive encoding and recall units. Critical points which make the processing of the structure differ from human processing concern the encoding and the storage: (1) The structure is built to massively store information instead of carefully selecting few impressions for storage in memory. (2) Reweighting of stored information due to changing constellations is not possible. Coming back to the example this would mean that a nice action at the beginning of the game has a strong impact on your categorization independent of what kind of actions - might they be impressive or not - followed afterwards. In this work we tackle these points through introducing an attention mechanism which drives the encoding and the storage of the structure. We analyse the model behavior in category learning tasks.