TY - JOUR
AU1 - Orlov, Tanya
AU2 - Yakovlev, Volodya
AU3 - Amit, Daniel
AU4 - Hochstein, Shaul
AU5 - Zohary, Ehud
AB - Abstract Serial memory is the ability to encode and retrieve a list of items in their correct temporal order. To study nonverbal strategies involved in serial memory, we trained four macaque monkeys on a novel delayed sequence-recall task and analysed the mechanisms underlying their performance in terms of a neural network model. Thirty fractal images, divided into 10 triplets, were presented repeatedly in fixed temporal order. On each trial the monkeys viewed three sequentially presented sample images, followed by a test stimulus consisting of the same triplet of images and a distractor image (chosen randomly from the remaining 27). The task was to touch the three images in their original order, avoiding the distractor. The monkeys' most common error was touching the distractor when it had the same ordinal position (in its own triplet) as the correct image. This finding suggests that monkeys naturally categorize images by their ordinal number. Additional, secondary strategies were eventually used to avoid distractor images. These include memory of the sample images (working memory) and associations between triplet members. Further direct evidence for ordinal number categorization was provided by a transfer of learning to untrained images of the same ordinal category, following reassignment of image categories within each triplet. We propose a generic three-tier neuronal framework that can explain the components and complex set of characteristics of the observed behavior. This framework, with its intermediate level representing ordinal categories, can also explain the transfer of learning following category reassignment. Introduction Serial memory is the ability to store and retrieve a list of items in their correct temporal order, an ability we share with other primates and non-primate species. However, the nature of the mental representation that allows retrieval of list items is still unclear. Is list memory better characterized as a set of associations between adjacent items& —& ‘chaining’ (Ebbinghaus, 1964), or is it based on symbolic associations between each item and its ordinal position (Ebenholtz, 1972)? Human subjects tend to assign names to items and generate associations among names or between names and ordinal positions. But monkeys and even pigeons are also able to reproduce ordered lists of arbitrary stimuli (Terrace, 1986, 1987, 1993; D'Amato and Colombo, 1988, 1989, 1990; Swartz et al., 1991; Terrace et al., 1995; Chen et al., 1997). We study here pre-verbal mnemonic strategies that are used to represent lists. In principle, working memory could be utilized to recall each stimulus' identity and its temporal position in a sequence. Working memory is advantageous in terms of generalization: it would be equally successful with both novel and familiar stimuli, as well as with randomly ordered images. Indeed, working memory is highly effective when monkeys are required to report whether a probe item was included in a previously presented list of up to 20 items. In fact, their performance is almost as accurate as that of humans (Sands and Wright, 1980). But remembering image temporal position is much more demanding. Performance deteriorates drastically when monkeys are required to choose which of two images appeared earlier in a list of five arbitrarily chosen items (Gower, 1992). When images are repeatedly presented in a fixed temporal order, subjects can use long-term memory strategies, as well as working memory. One available strategy is to generate an association between adjacent images in the fixed sequence. Monkeys are highly skilled in generating such paired associations (Sakai and Miyashita, 1991; Murray et al., 1993; Naya et al., 1996; Erickson and Desimone, 1999). Using a series of paired associations, a chain of images can be recalled. In fact, monkeys trained on a series of images successfully report the order of a random pair from the fixed sequence, even non-sequential pairs (Terrace, 1987; D'Amato and Colombo, 1988; Swartz et al., 1991). Similarly, monkeys trained on a sequence of pairs (A–B, B–C, etc.) successfully report the order of any pair, even non-sequential pairs (McGonigle and Chalmers, 1992; Treichler and van Tilburg, 1996). In both cases, analysis of response times suggests an internal serial recap of the list, perhaps on the basis of serial pair associations. Another possible mnemonic routine is memorizing the ordinal position of each image. In a recent study, monkeys were trained on four lists, each containing four novel photographs of natural objects (Chen et al., 1997). The task was to touch the simultaneously presented images in the correct order (A1– A2–A3–A4, B1–B2–B3–B4, C1–C2–C3–C4, D1–D2–D3–D4). When the monkeys had mastered this task, the items were shuffled, taking one item from each list, so that in two derived lists the ordinal numbers of the items were maintained (e.g. A1–D2–C3–B4) while in two others they were not (e.g. B3– A1–D4–C2). Lists with maintained ordinal position were acquired rapidly and virtually without error, while derived lists in which the ordinal position was changed were as difficult to learn as novel lists. This pattern of transfer to derived lists implies that the monkeys originally acquired some knowledge about each item's ordinal position, rather than only generating a chain of serial pair associations for each list of items. In the present study we introduced an experimental paradigm in which monkeys could use all three of the above-mentioned strategies: ordinal categorization, working memory and paired/ chained associations. This allowed us to assess which strategy is the most natural, in that it is the first to be learned, and which is dominant, in that it has the biggest effect on performance. In addition, we were able to measure specific characteristics of each of these strategies enabling us to suggest a model incorporating all of them. Some of the results presented here have been published earlier in brief form (Orlov et al., 2000). Materials and Methods Subjects and Apparatus Four adult male macaque monkeys, two Macaca mulatta, (J, 7.4 kg; R, 9.2 kg) and two Macaca fascicularis (S, 4.3 kg; G, 4.7 kg) participated in the experiments. Experiments were performed in accordance with NIH and Hebrew University guidelines for use of laboratory animals. We used operant conditioning to train the animals on a memory task, giving a fruit juice reward during training sessions. The monkeys sat in a primate chair 30 cm from a computer color monitor equipped with a touch screen (31.5 × 26.5 cm). Trial events, stimulus presentation and data recording were computer controlled. Behavioral Shaping Four adult male macaque monkeys, two Macaca mulatta, (J, 7.4 kg; R, 9.2 kg) and two Macaca fascicularis (S, 4.3 kg; G, 4.7 kg) participated in the experiments. Experiments were performed in accordance with NIH and Hebrew University guidelines for use of laboratory animals. We used operant conditioning to train the animals on a memory task, giving a fruit juice reward during training sessions. The monkeys sat in a primate chair 30 cm from a computer color monitor equipped with a touch screen (31.5 × 26.5 cm). Trial events, stimulus presentation and data recording were computer controlled. Behavioral Shaping The monkeys learned to press a lever following onset of a fixation spot and not to release it during presentation of three sequential sample stimuli. At first, the images in each triplet of sample presentations were chosen randomly (without repetitions) from the entire set of 30 images. During the test phase of the trial, the same three images were displayed on the screen simultaneously. This served as a ‘go’ signal for the subjects to release the lever and touch the images on the screen. A touch of each image made in any order was rewarded, but a repeated touch of an image that was touched previously in the same trial was considered an error and terminated the trial. During a brief final period of shaping (monkey J, 100 trials; S, 400 trials; R, 700 trials; G, 800 trials) a distractor image (not included in the sample triplet) was added to the test stimulus. The monkeys were still rewarded irrespective of the order of touches, as long as they avoided repeated touches and touching the distractor. The monkeys learned to press a lever following onset of a fixation spot and not to release it during presentation of three sequential sample stimuli. At first, the images in each triplet of sample presentations were chosen randomly (without repetitions) from the entire set of 30 images. During the test phase of the trial, the same three images were displayed on the screen simultaneously. This served as a ‘go’ signal for the subjects to release the lever and touch the images on the screen. A touch of each image made in any order was rewarded, but a repeated touch of an image that was touched previously in the same trial was considered an error and terminated the trial. During a brief final period of shaping (monkey J, 100 trials; S, 400 trials; R, 700 trials; G, 800 trials) a distractor image (not included in the sample triplet) was added to the test stimulus. The monkeys were still rewarded irrespective of the order of touches, as long as they avoided repeated touches and touching the distractor. In the basic behavioral task, (described below), sample images were shown as triplets of fixed order and the image touches were only rewarded if they were made in the order of the sample sequence. The dynamics of learning this task and the resulting functional structure are the main focus of this study. Stimulus Images We used 30 fractal images (Miyashita et al., 1991), divided into 10 constant non-overlapping triplets and presented in a fixed temporal order from A to J (A1, A2, A3; B1, B2, B3; … ; J1, J2, J3; A1, A2, A3 … ; see Fig. 1a). Each trial consisted of a triplet of three sample stimuli, with each image having a fixed ordinal position in its triplet (1st, 2nd or 3rd). During each trial, a given triplet was displayed twice: first, sequentially, during presentation of the sample sequence and then, simultaneously, during the test presentation. A distractor image, randomly chosen from the remaining 27 images, was added to the test presentation of the triplet, as demonstrated in Figure 1b. Each image, including the distractor, has a fixed ordinal position in its own triplet. Images A1–J1 were always shown as the first sample of the trial, A2–J2 always second and A3–J3 always appeared last. So, images were also divided into three abstract categories according to their ordinal position in their triplet. We ask if the monkeys used this information to carry out the task. Experiment I& —& Basic Behavioral Task The testing paradigm for the basic task is demonstrated schematically in Figure 1b. Following presentation of a fixation spot on the screen, the monkey pressed a lever. This initiated presentation of a sequence of three sample stimuli (a triplet) in a fixed temporal order. Each sample image was shown at the center of the screen for 500 ms with a 1 s inter-stimulus interval. Finally, the same three images were presented together with a fourth one, a distractor stimulus, at random positions on the screen. This test display was a go signal for the monkey to release the lever and touch the three images of the sample sequence, in the order of their prior presentation, without touching the distractor. Touching a wrong image (an image out of order or the distractor image) terminated the trial& —& and the next trial with the following triplet was initiated& —& while each correct touch was rewarded with juice. The third correct touch completing the trial was rewarded with a larger portion of juice. The monkeys were allowed to perform as many trials as they wished in a given session and the number of trials per session varied considerably from session to session (17–269 trials; 1 session per day). Experiment II& —& Task Variations for Strategy Identification following Training on the Basic Task As outlined in the Introduction, we are interested in the role of different memory strategies in task performance: categorization of images according to their position in their triplet; stimulus–stimulus associations; and working memory of viewed triplet images. We are also interested in the formation of these strategies during the process of learning. The approach used was to test task performance while disabling access to one or two of the possible mnemonic routines and to vary the learning effect by presenting variations in the stimulus set or task structure. To this end, we introduced several variations of the task and structure of the stimulus set. (a) Trials with no samples (Fig. 2, ‘no samples’, monkeys J and R): in this task variant, touch-order performance was tested in the absence of sample stimuli, thus eliminating the effects of sample working memory. The sample sequence of three images was replaced by a sequence of three gray rectangles, to maintain the temporal structure of the trial. (b) Trials with images shuffled among triplets, with each image preserving its original ordinal position (categories maintained; Fig. 2, ‘shuffled’, monkeys J and R): in this variant, images within each category were shuffled to form new triplets for each trial. The shuffling maintained the ordinal category of each image, but destroyed the internal membership of the integral triplet, thus eliminating the effects of intra-triplet associations. The same shuffled triplets of images were presented in both the sample and test stimuli of the trial. Note that, although formally the above manipulation abolished all visual stimulus–stimulus associations, including both intra- and inter-triplet associations, the inter-triplet associations proved irrelevant& —& see (e) and results& —& thus we will refer to this manipulation from now on as abolishing the intra-triplet associations. (c) Trials without samples and with images shuffled among triplets (Fig. 2, ‘no samples + shuffled’, monkeys J and R) combined the two previous manipulations; the shuffled triplets of images were displayed only during the test presentation. Reward was contingent only on touching the correct category. (d) Images scrambled within each triplet (monkeys J and S): the order of presentation of the sample sequence of stimuli was randomized from trial to trial (e.g. A3, A1, A2; B2, B1, B3; C1, C3, C2; . . .) disregarding the trained categories for the triplet images, but the reward was given for response touches in the originally trained order, i.e. according to the learned fixed categories. This variant was introduced to test the effect of the sample sequence on touch order and, separately, on discrimination of the distractor. (This is the only case where reward was not strictly according to the presented sample sequence.) (e) Trials with shuffled order of triplets (monkeys J and R): in this version of the task we manipulated the order of triplet presentation, maintaining the triplets intact. This allowed us to assess the effects of inter-triplet associations. Experiment III —& Task Variations for Identification of Learning Mechanisms (a) Retraining with reassigned categories (monkeys J and S): the three categories were rearranged according to a cyclic permutation& —& [I, II, III] became [II, III, I]. Images that were shown second originally, were now shown first, etc. (see below, Figs 5 and 6, shift 1). The specific image from those of its category that was presented in each new triplet was chosen at random, as in the ‘shuffled’ paradigm (b) above. Following training and testing with these new sequences, a second reassignment was introduced (shift 2), which was the same permutation performed on the new triplets, and the monkeys were retrained again. (b) Learning transfer: to test whether category assignment was a property of each single image, or if a common label was assigned to the entire set of images of the same category, we tested learning transfer, as follows. Training was initially carried out on half of the newly generated triplets (A, C, E, G and I for monkey J; B, D, F, H and J for monkey S). The reversed choice of subsets was intended to preclude effects due to particular image features. When performance on these subsets of triplets approached asymptotic levels, we tested the degree of learning that was transferred to the remaining five triplets. Thus, training was done four times: shift 1, 1st half of triplets; shift 1, 2nd half; shift 2, 1st half; shift 2, 2nd half. In each case samples were shuffled among five images of the same category from the appropriate half of the triplets. For shift 1, samples were shown for each trial, for both the 1st and 2nd half. For shift 2, monkey J was trained again with sample sequences for the 1st half of the triplets, but tested on the 2nd half without samples. Monkey S was not shown any sample images for either half of shift 2. Training on shift-without-samples serves also to probe the possibility of learning categories based on the test stimulus alone. Results and Interpretation Experiment I: Ordinal Number Categorization as the Dominant Mnemonic Strategy Results To study the nature of the routines that the monkeys used while learning the basic task, we analysed the distribution of their choices, sorted by touch number within the trial and by the category of the distractor added to each test display. Note that within its own triplet, the distractor had a fixed ordinal position (1st, 2nd or 3rd). This is hypothesized to determine its ‘category’& —& and degree of interference in trials where it acts as a distractor. The dependence of the results on this categorization will uphold or reject the hypothesis that the monkeys use temporal order categorization to perform the task. Figure 3 shows the development of correct and erroneous distractor choices made during the three consecutive touches, averaged across four monkeys. At first, the monkeys touched the images in random order so that each image had a 25% chance of being touched. Beyond ~100 trials, the monkeys rapidly learned to touch the correct item, excluding distractor choices when the distractor belonged to a different ordinal category than the correct image. When the distractor was from the same category as the correct item, touches were roughly evenly divided only between the two; the sum of the touches of the correct image and the distractor added up to nearly 100% (dashed line). These are direct indications that categorization was the major and natural strategy used by the monkeys for this task. In all cases, differentiation between the correct image and the distractor was earlier when the distractor was from a different category than when it belonged to the same category as the correct image (paired t-test, P < 0.00001, n = 12). Interpretation Had the monkeys used working memory, recalling the information contained in the sample sequence, they could have avoided the distractor altogether, irrespective of its ordinal category. Similarly, had strong associations between the images belonging to a given triplet been formed and used, they could also have avoided the distractor. The strong dependence on the ordinal position of the distractor image suggests that the monkeys established long-term memory associations of each image and its fixed ordinal position number. Further experiments are needed to determine the length of sequence that may be represented in this way; preliminary results indicate that monkeys have no problem with four-item lists. Note that the information needed for establishing ordinal number categories could have been extracted either from the sequential presentation in the sample stage of the trials or, on the other hand, from the rewarded sequence of touches in the test stage of each trial. The findings of Experiment I lead us to propose the existence of ‘category neurons’ (see General Discussion). We suggest that these cell populations may be excited by the serial position of an image in the sample sequence and/or the ordinal position of the anticipated touch (or reward number) during the test phase of the trial. With training, they become associated with the images of a specific category. Experiment II: The Contribution of Alternative Memory Strategies Results The dependence of performance on distractor category suggests that the monkeys naturally and rapidly internalize strong connections between the images and mnemonic representations of distinct image categories, classified by their ordinal position. But categorization alone can only lead to 50% success when the distractor and the correct stimulus belong to the same category. During later stages of training, the monkeys did indeed learn to avoid the distractor. Working memory, transmitting the visual content of the sample images, or intra-triplet associations, indicating which images belong to a proper triplet, are two possible strategies that could be used to separate out the distractor. (Inter-triplet associations proved to be irrelevant for task performance. Performance was just as good as in the basic task when the order of triplets was shuffled, maintaining the integrity of the individual triplet structure; ANOVA, F = 0.04, P = 0.84.) We proceed to evaluate the role of the two memory strategies, after performance neared asymptotic level, using task variations that precluded one, or both, as demonstrated in Figure 2. Performance was tested in the absence of the sample sequence of images on each trial (the ‘no samples’ paradigm). This eliminated a potential reliance on working memory. When the distractor was from a different category than that to be touched, performance was nearly unaffected (correct choice remained >80% and choice of distractor was <10%; Fig. 4, ‘no samples’, different: gray bars). Only when the correct image and the distractor belonged to the same ordinal category was performance degraded so the distractor was chosen more often& —& but never as often as the correct image (Fig. 4, ‘no samples’, same: striped bars). A similar pattern was observed when images were shuffled within each category, disabling associations between triplet members (Fig. 4, ‘shuffled’). Here, too, performance was unaffected in the ‘different’ trials and was degraded, but not to chance level, for the ‘same’ trials. Jointly, these results indicate that: (i) both working memory of the sample and intra-triplet associations are used in differentiating between the correct image and the distractor and (ii) neither of these mechanisms suffices; they must be used in tandem with each other and with the monkey's knowledge of each item's ordinal position to produce best performance. When the two manipulations described above (‘no samples’ and ‘shuffled’) were combined, the monkeys performed just as well as in the basic task, as long as the category of the correct item and distractor differed (labeled ‘different’ in Fig. 4). This finding indicates that when the correct item and distractor differed in ordinal category, memory of the samples and intra-triplet associations provided little, if any, additional benefit in performance. Alternatively, when the correct item matched the category of the distractor (labeled ‘same’ in Fig. 4), choice was distributed equally between the two items (Fig. 4, ‘no samples + shuffled’ – ‘same’). Thus, these two strategies (working memory and associations) suffice to provide a complete account of the improvement in performance above the levels reflecting use of ordinal category alone. ANOVA revealed that significant factors in performance variance were: distractor category (same or different, F = 155, P < 0.00001); task type (‘basic task’, ‘no samples’, ‘shuffled and no samples + shuffled’, F = 11.1, P < 0.00001); and touch number (F = 5.8, P < 0.01), but not monkey identity. The only significant interaction term was distractor category * task (F = 5.8; P < 0.01)& —& reflecting the invariance of performance across tasks when the distractor was of a ‘different’ category, but not when it was of a ‘same’ category. Together these factors accounted for 91.5% of the total performance variance. Next we asked: do monkeys rely on working memory only for the identity of the images shown in a given trial or do they also encode in this way their ordinal position? To answer this question, we briefly changed the task paradigm in yet another way. The order of presentation of the sample stimuli was randomized from trial to trial, but the reward was given for response touches in the original order, i.e. according to the learned fixed categories. Thus, in one trial the triplet may have been shown in its original learned order& —& A1, A2, A3& —& while in another trial the order may have been A3, A1, A2. The required response, however, was fixed at A1, A2, A3. If only image identity is encoded in working memory, the monkeys should not have a problem with this revised paradigm. If, however, both image identity and ordinal position are stored in working memory, a conflict will result between serial information from working memory and category information from long-term memory. Monkeys S and J were tested with these scrambled sample sequences (not to be confused with ‘shuffled’ triplets). Performance was essentially unchanged by such scrambling, in both the same (ANOVA, F = 0.003, P = 0.96) and the different (ANOVA, F = 0.24, P = 0.64) cases, which indicates that ordinal category was not stored in working memory, at least under the present circumstances. Interpretation Nearly perfect performance, irrespective of task manipulation, was obtained when the distractor belonged to a different category than the correct image (Fig. 4, gray bars). This adds strong support to our previous conclusion that the main and first mnemonic routine assimilated by the monkeys was retrieval of ordinal categories from long-term memory. Sample-image working memory and long-term, intra-triplet associations between the images provided additional cues for differentiating between the distractor and correct image in same trials. Note that the fact that for a substantial training period (a few hundred trials, Fig. 3) there is no effective segregation of the distractor, when it is from the same category as the correct image, indicates that these strategies are not initially available. As noted above, these results suggest that working memory was used only to recall the presence of images in the sample sequence and not their ordinal position. When both mnemonic short-term and long-term routines are available, and the task requires memory of the temporal order of image presentation, our monkeys relied on long-term memory strategies rather than on working memory. Establishing categories according to ordinal position required a shorter training period than engaging working memory of the images or associations between sequential items. Neither was necessary to encode the sequential order in our task. The ordinal position of each image was learned without chaining, i.e. without knowledge of the identity of its temporal neighbors. It has previously been suggested that ‘delay-period activity’ may underlie working memory and that correlations among neuron populations manifesting such activity may be the cause of associations among images. We refer to theoretical models of these physiological phenomena in the General Discussion. Experiment III: Learning and Generalization Following Category Reassignment We re-trained the same monkeys following category reassignment of the same 30 stimuli according to a cyclic permutation, i.e. [I, II, III] to [II, III, I], as described above: all images of the second category were now shown first, third category images second, etc. In addition, new triplets were constructed for each trial by shuffling the images within each category independently, to eliminate potential reliance on prior intra-triplet associations (as in the ‘shuffled’ paradigm of Experiment II). Reassignment training was carried out first for one half of the triplets, then for the second half. The cyclic reassignment was carried out twice. For these reassignment tests, a distractor was not introduced into the test stimulus. Testing following reassignment had two goals. The first goal was to study whether the balance between short- and long-term mnemonic routines, which was established in our basic task, was also maintained during learning of a new category order. We found above that when there was a conflict between working memory and categorization, performance reflected long-term category memory. However, there reward was administered according to the old, long-term memory categories. Here, instead, reward is given not according to the previously trained categories, but according to the new order of the samples. If working memory plays a dominant role, adjustment to the new temporal order should be instantaneous because the sample is available to the monkeys on each trial. On the other hand, if long-term memory of the image categories is the dominant mnemonic routine, a long learning period would be needed to assign a new response to each of the image categories according to their new phase. In addition, since working memory is established later than categorization in initial learning (see above, Fig. 3), it may play a more dominant role when training a reassignment, i.e. following its use for these images with their original category assignment. The second goal of training for new categories was to obtain insight into potential generalization of the categorization mechanism. We trained the new categories only on a partial set of triplets. If there is transfer to untrained images, the correct responses for the second set of triplets should be learned faster than the first. Moreover, if there is transfer, we may conclude that all images belonging to a category share the same label. Results As to the balance between categorization and working memory, the results, shown in Figure 5, were unequivocal. The monkeys responded at first according to the previously learned categories rather than the current order of sample images: they tended to touch first (left column) images belonging to the previous first category (black lines), rather than the current category (hatched lines). Interestingly, already in the first session, if the monkeys had made a correct choice in their first touch, they made few errors in their second touch (middle column). This may derive from the fact that the new 1st and 2nd images belonged to the 2nd and 3rd categories in the previous situation. Thus, the sequential order of these two categories remained intact (former categories 2nd and 3rd; e.g. images E2 and A3 in the example shown in Fig. 5). On the other hand, the current 3rd category was 1st in the previous setting, so touching last the 3rd image (e.g. C1) after the 2nd current image (A3) does not follow from the previously learned sequential order. In fact, monkey J made more errors on its third touch than on its second touch, even though simple elimination of the already touched images would have led to perfect performance. Since the monkeys were trained with the reassigned categories on only half of the triplets, we were able to probe the degree of generalization of learning to the other (untrained) members of the same reassigned category. Note that although touch order was changed, category membership remained intact: all images of a given category required the same touch response. If all images belonging to a category share a ‘category label’, then learning the second subset should be accomplished more rapidly than the first. Figure 6 shows the learning dynamics of the first five triplets (full curve) and second five (hatched curve) for each monkey. The curves are superimposed to facilitate comparison. The results clearly indicate that learning of the new categories is partially transferred: both monkeys learned the second half more rapidly than the first, though some relearning was required. The effect is concentrated in the first session. The insets show first touch performance level as a function of the trial number in the first session. They provide comparison of the learning dynamics on a finer temporal scale: learning the first subset (full curve) and the second (hatched curve). As early as the first session, learning is more rapid for the second five triplets, although the initial offset may not explain the entire difference in performance. It may be proposed that the source of the more rapid learning on the second half of the images stems from the monkeys learning to rely more on working memory of the sample images than on the categorization strategy. To test this hypothesis, the transfer test was repeated, once more reassigning the categories by cyclic permutation and again reshuffling the images in each category (shift 2). Whereas Monkey J was again trained with sample sequences for the first subset of triplets but without them for the second subset (Fig. 6, top right graph), Monkey S was trained for the second category reassignment without presentation of samples for either the first or second subsets (bottom right graph). Thus, we eliminated working memory of the samples as a strategy for assimilating the reassignment. The results were very similar to those obtained in the first reassignment, i.e. faster learning but only partial transfer. Moreover, asymptotic performance was insensitive to whether the sample stimuli were presented at all. We do not have sufficient data to judge whether there was a difference in learning speed with and without samples. We conclude that better performance with the second half of the images was not due to a shift in the balance of the mnemonic strategies used. Furthermore, the monkeys were able to learn the reassignments even when the sample images were not presented, i.e. learning may result exclusively from experience with the test stimulus. Interpretation When categories were reassigned (creating new triplets out of the same items), we found the following. In early phases of retraining the monkeys tend to select touch order according to the previous assignments, indicating that, at least initially, working memory of the sample stimuli does not determine touch order, when in conflict with long-term memory of ordinal categories, even when the reward is scheduled according to the working memory information. The monkeys can learn the reassignments even if sample image sequences are not presented in training, i.e. learning may result exclusively from operation on the test stimuli. When learning the reassigned categories on a subset of the triplets, there is a partial transfer of this information: learning is accelerated for the remaining subset of images. The monkeys acquired knowledge of the sequential order of the abstract categories. This knowledge was not due to individual image associations (chaining), since the specific images shown were randomized from trial to trial, with images shuffled within categories. The sequential order of category touches may have been based on associations between abstract categories. Taken together these results indicate that monkeys are able to organize perceptually non-similar images into classes and manipulate them according to changing task requirements. The main conclusion from the observation of partial transfer is that assignment of ordinal number categories in this experiment is general and abstract, not a label associated with each image, separately. Otherwise, no transfer would have been possible: the monkeys would have had to learn the new category of each image separately. In principle, the monkeys only needed to learn a new re-mapping of the behavioral output (ordered touches) for the fixed categories. We implement this in the model by a distinct level of representation, relating to a planned sequence of motor responses, which is directed by the category label neuron populations (see General Discussion). Model Framework and General Discussion The experiments described in the present study indicate that in a visual serial recall task monkeys assimilate first and very rapidly the ordinal position categories of the images. In other words, they quickly learn to recognize which images (10 in each group) are first, second or third in their corresponding triplets. The introduction of a distractor, itself with a given ordinal position category, allowed us to establish that: working memory and intra-triplet associations enter the monkeys' behavioral repertoire significantly later than categorization; working memory and intra-triplet associations serve as secondary mechanisms, to separate the distractor from an image of the same ordinal category; these two additional mechanisms are not involved in determining the touch order, but only which three images to touch and which one to avoid. We now link these findings to a picture of memory processes that stems from neural population concepts. According to our view, images are represented and recalled by distinct patterns of activity across a population of neurons. These patterns of activity are shaped and determined by synaptic interactions that are formed in the course of repeated presentations of specific visual images. With enough repetitions, ongoing ‘delay’ activity may develop in specific populations in localized cortical regions (Amit et al., 1994; Amit, 1995; Amit and Brunel, 1997). The persistent delay activity is maintained by recurrent synaptic feedback between interconnected neurons within a local module, built up as stimuli become familiar. The memory process is initiated by presentation of the visual stimulus, which generates a pattern of response across the neuronal population. Following removal of the visual stimulus, due to the feedback connections within the neuronal population, the dynamics of the network are such that it settles into a stable state (an attractor). In this state, most neurons are firing at their spontaneous level, but some distinct groups of neurons continue firing at elevated levels although the visual stimulus is no longer present. Since each visual stimulus evokes a characteristic pattern of delay activity, the delay activity distribution is the neuronal engram of the last familiar stimulus seen. The distributed nature of the representation allows storage of a large number of patterns (i.e. stable delay activity distributions) in the same neural module, by the same synaptic structure. This framework can also explain the generation of associations between stimuli repeatedly presented in temporal proximity (as is the case for intra-triplet associations). Neurons that are part of an attractor representing one stimulus will remain active during the delay period until presentation of the next stimulus. The joint activity (in a time window of tens to hundreds of milliseconds) will allow for Hebbian strengthening of the synapses between neurons belonging to the two populations (Hebb, 1949). If stimuli are systematically viewed in a fixed temporal order, this Hebbian learning will eventually lead to similar mnemonic representations (i.e. patterns of firing rates) for the two stimuli. Thus, an associative memory will be formed. Recent theoretical and experimental data suggest this is plausible (Yakovlev et al., 1998). It is important to note that the stable attractors (‘selective delay activities’) are formed during a slow learning process that shapes the synaptic structure between the network members. Associations between temporally adjacent images require even more repetitions, as they rely on the prior establishment of the individual attractors. These considerations could explain why working memory and intra-triplet associations are utilized late in our behavioral paradigm. Alternatively, associations could be also formed during the test phase of the task when the three images appear together. What is the neural basis of ordinal number categorization? Any attempt to model this cognitive process must take into account a number of key features we have found: categorization occurs rapidly, well before working memory and intra-triplet associations are utilized (Fig. 3); categorization does not rely on creating chains between sequential images (Fig. 4, ‘shuffled condition’); following category reassignment, learning takes place even when sample image sequences are not presented in training, i.e. learning can be exclusively based on touch order of the test stimuli (Fig. 6, bottom); when learning the reassigned categories on a subset of the triplets, there is a partial transfer of the new ‘knowledge’& —& learning is accelerated for the remaining subset of images (Fig. 6, dashed versus solid). We propose a general framework, based on three levels of neural population activity (Fig. 7), as follows. Population activity representing abstract visual images (A1–J3 in Fig. 7) — groups of neurons with selective visual response. Repeated activation of such populations may lead to the establishment of selective delay activity and provide the basis for working memory and intra-triplet associations. Population activity representing classification of ordinal number. (I, II, III in Fig. 7) — groups of neurons whose activation is task-context dependent. These neurons may be excited either by the serial position of an image in the sample sequence or by the ordinal position of the anticipated touch during the presentation of the test stimulus. Thus, these populations represent abstract ordinal number categories. Population activity representing the ordered sequence of a planned touch (motor) response (1st, 2nd, 3rd in Fig. 7) — neuron groups whose activity reflects a decision concerning the order of the motor response. Other types of task-related population activity that aid shaping behavior are neuronal populations related to the ‘go signal’ and neural populations activated by the administration of the reward. We first sketch the use of such neuronal elements in forming a representation of ordinal number categories. The neurophysiological evidence and loci of all types of neuronal elements introduced here will be discussed below. In the process of training for the basic task, above, each image population becomes associated with a neural group that represents a particular category (ordinal position in a triplet). The potentiation of synaptic interconnections between these neural populations results from independent activation, during presentation of a given sample image, of both the neurons that are selective to that image and those selective to its ordinal category (i.e. the sample ordinal position). The concurrent, direct activation of these two neural groups leads to rapid potentiation of synapses between these assemblies, leading to enhanced activation of the category label neurons when an image is detected in the test stimulus. As we saw, category-learning can be extracted from the test stimulus alone, i.e. even in the absence of sample images. If category cells represent an abstract concept of order, these cells might be activated not only by the serial order of the samples, but also by the serial order of the expected touch. Category cell populations become activated in turn by expecting performance of a specific touch (1st, 2nd or 3rd). In other words, during the test phase, a given touch will be associated with a given motor-decision population activity. Simultaneous activity in the two populations will lead to potentiation between the category and the motor decision neurons. That potentiation will be more effective when the touch is correct and reward is administered. Upon presentation of the test stimulus there is also a ‘go’ signal& —& initiated by the multiple image stimulus. At this point, specific category neurons receive direct activation due to the expected touch number as well as indirect activation via image-to-category afferents. These strongly activated category cells allow selection of a specific touch response via the motor-decision population. Note that associations between abstract categories may reflect establishing of lateral connections between category label populations. The three-tier framework for ordinal number categorization, with its intermediate level representing categories, is able to explain the observed transfer of learning following category reassignment to untrained images. The connection between the category-label representation and the motor-decision population is common for all images in a given category and can underlie the generalization (or transfer) to other images of the same category. According to this framework, following category reassignment and training with half the images, a re-mapping takes place between the category label populations and the motor-decision neurons encoding the required touch order (Fig. 7, dashed arrows). Therefore, when encountering other images belonging to the same category during the category reassignment training of the second half of the images, the same touch behavior is expected, demonstrating generalization of learning. However, the transfer we found was not complete. This partial effect requires some amendments to the proposed framework. These might include initial learning affecting image population to category population connections and/or direct connections from image populations to response populations. The former is expected to occur& —& and ultimately prevail& —& since the revised order of sample presentation or expected touch would automatically activate the appropriate category population. This framework is meant only to capture the essential features of categorization learning we have observed. An elaborated neuronal model is beyond the scope of this paper and will be presented elsewhere. Neurophysiological Evidence for the Model Framework What is the neurophysiological evidence for the neuronal elements introduced in our framework and where in the brain can these neural populations be found? Image representations are thought to be located in inferotemporal (IT) cortex, especially in the perirhinal (PR) area. This area is implicated as a crucial element for long-term memory storage of visual objects (Mishkin, 1982; Meunier et al., 1993; Murray et al., 1993; Tang et al., 1997; Thornton et al., 1997; Buckley et al., 1997; Buckley and Gaffan, 1998a,b,c; Parker and Gaffan, 1998; Buffalo et al., 1999; Miyashita and Hayashi, 2000; Erickson et al., 2000). Single neuron recordings in the anterior IT cortex revealed that neurons in these areas have delay activity that is stimulus-specific and therefore may be related to mnemonic processes. This delay activity may reflect activation of a long-term memory image trace (Amit, 1995; Amit and Brunel, 1997). Neuronal populations in PR cortex are therefore likely to participate in the image populations suggested in the model. Their delay activity following a specific test stimulus (in a Delayed Match Sample task) is maintained throughout the inter-trial-interval until the next sample stimulus in the following trial (Yakovlev et al., 1998). Therefore, it can link the representations of temporally adjacent images and could work as a vehicle to generate long-term visual associations (Miyashita, 1988; Sakai and Miyashita, 1991; Naya et al., 1996; Yakovlev et al., 1998; Erickson and Desimone, 1999). The intra-triplet associations in our experiment may have been learned via generation of correlated delay activity of this kind. Stimulus-specific delay activity has also been recorded from neurons in prefrontal (PF) cortex (Fuster, 1973; Miller et al., 1996) and entorhinal (ER) cortex (Suzuki et al., 1997). This activity can survive during presentation of another stimulus (e.g. a stimulus intervening between sample and match stimuli), while IT delay activity can not (Miller et al., 1996; Suzuki et al., 1997). Thus, PF and ER delay activity may support working memory performance in our task, requiring simultaneous memory of multiple images in order to segregate the samples from the distractor. At the motor response end, the role of supplementary motor area (SMA) and pre-SMA are critical for planning and execution of movement sequences. These areas are particularly prominent in retrieving information from memory for task completion (Tanji and Shima, 1994; Picard and Strick, 1997; Shima and Tanji, 1998), suggesting that these could be our motor decision population neurons. Somewhat surprisingly, possible candidates for ordinal category neurons suggested in our scheme may be found in various cortical areas. M1, pre-SMA, SMA, cingulate motor area (CMA), superior parietal cortex, striatum and frontal eye fields (FEF) all contain specific neurons, which may encode the ordinal position of hand or eye movements, when a sequence of such movements is required (Barone and Joseph, 1989; Kermadi and Joseph, 1995; Clower and Alexander, 1998; Carpenter et al., 1999; Procyk et al., 2000; Shima et al., 2000). The ordinal position of the target (during fixation) or the hand movement during a sequence of movements was the main parameter affecting the activity of these neurons. This selectivity to ordinal position was maintained irrespective of the other reach movement parameters (direction, kinematics) or target spatial locations. More specifically, Joseph and colleagues recorded neuronal activity in FEF and the caudate nucleus while monkeys performed a sequential reaching movement task, using visual targets (dots) presented in a sequence (Barone and Joseph, 1989; Kermadi and Joseph, 1995). The monkeys were required to remember the target locations in their presented order and execute a series of reach movements in the same order. A significant proportion of the cells in FEF and the caudate nucleus responded only if the corresponding target came first in the sequence, irrespective of its position on the screen; other cells responded only if the target was second or if it had complex time relationships with the other targets. Similar neuronal activity was found more recently in the CMA (Procyk et al., 2000). In all the above tasks, the ordinal position of a specific target location varied from trial to trial. Therefore the monkeys had to rely on working memory to perform the task correctly. However, neuronal activity, reflecting the numerical order of movement elements during retrieval of learned sequences from long-term memory, was found in SMA and pre-SMA (Clower and Alexander, 1998). Surprisingly, serial order effects were recently found also in primary motor cortex (M1) representing the hand area. In this study, up to five visual targets were presented successively on a circle and then the color of one of them was changed. The monkeys had to execute a hand movement towards the next target in the sequence. About half of the neurons were selective to the serial position of the target, irrespective of its spatial location (Carpenter et al., 1999). Thus, multiplexed encoding of both direction of movement and serial position order is common in M1. These studies suggest that the ordinal category neurons may reside in areas closer to the motor output. However, in all these cases, the targets were defined by their serial position and location in space. In contrast, in our case, the target identity varied with serial position, while the location in space was irrelevant. Therefore, categorization was based on the ordinal position of objects. Accordingly, our ordinal category neurons must receive direct input from object selective neurons, to become selective to specific objects (within a category), with repeated experience. Thus, our ordinal category neurons are expected to be more closely linked with visual areas processing object information than the above populations. Neuronal populations with strong selectivity for objects within a category were found recently in IT cortex (Tomita et al., 1999). In this study, monkeys were trained to memorize visual stimulus–stimulus associations among 20 cues and five choice images. The cue pictures were randomly sorted into five categories. Thus, each of the four cues in a category specified a common choice. Some neurons in IT had category-selective delay activity: it was enhanced following all cues of a specific category, but not to cues from any other category. Choice responses were also strongest for this category. To summarize, mnemonic information about visual objects is most likely conveyed by neuronal populations in IT cortex. Information about sequences is probably more related to the motor output and can be found in various motor planning areas, including FEF, pre-SMA (or SMA), CMA, superior parietal cortex and the striatum. Our putative ordinal category neuron populations would respond specifically to the serial position in the sequence of images and the expected touch number. Thus, they must receive an (object selective) visual input and be directly connected with motor planning populations. They are also modulated by a reward signal, perhaps originating from the amygdalar system. A possible site for ordinal category neurons is ER or PF cortex, which are strongly interconnected with both the motor and the object areas in the ventral visual pathway. Neurophysiological experiments are needed to identify sites of candidate neural correlates of the three hypothesized levels (im-age, category-label, motor-decision), as well as go/reward signals. Image Salience versus Ordinal Number Categorization Our behavioral task was designed in such a way that there was a premium on getting the first touch right, as every trial had a first touch opportunity while a wrong first touch aborted the trial without further touch, or reward. As mentioned above, the related salience of the images may have played a role in governing the monkeys' behavior in the early stages of learning. This salience hypothesis predicts that images that are rewarded more often would always be preferred over images that have a lower probability of reward. Alternatively, differences in image salience may result from primacy effects, i.e. earlier items in a list are better remembered then later ones (Buffalo et al., 1994). Thus, a favorable image would be more likely to be touched when presented as either a correct image or a distractor. Figure 8 displays the rate of (erroneous) touches of an image when presented as a distractor, as a function of the rate of correct touches of the same image (in trials when the image was included as a sample). Note that these rates were computed from different trials since a given image cannot serve as a target and a distractor in the same trial. In the initial stages of learning (Fig. 8, left graph) there was a clear positive correlation between the proportion of distractor and correct touches of an image (r = 0.90). Could this ‘salience rule’ explain serial order selection in our experiments? Direct image salience cannot explain our results. First, the monkeys clearly learn to avoid repeat touches not only of the same image, but also (distractor) images belonging to the same category and therefore of the same salience. Second, most of the correlation of Figure 8 stems from inter-category differences, while correlations within each ordinal category are weak (r = 0.11). Finally, the learning transfer observed relies on the establishment of ordinal categories and would not be expected if images were selected according to their individual salience. Can touch choice derive from ‘category salience’ (together with a ‘no category touch repetition’ rule)? Of course, category salience is correlated with ordinal position, so it may be difficult to differentiate between the two strategies. But at later stages (Fig. 8, right graph) the monkeys learn that each image can equally lead to a reward, if touched in the correct order. The correlation between the percentage of correct touches (salience) and the proportion of distractor touches vanishes. The salience hypothesis is also unable to explain why the monkeys immediately make a correct second touch following the cyclic shift in category assignment (Fig. 5, second column). The correct second touch involves selection of an image from the lowest salience category (previous 3rd), while the favorable alternative (1st) is available. Thus, ultimately, categorization must be based on image ordinal number. It remains to be seen if ordinal number categories can evolve in other training conditions, in which the reward contingencies of all images are identical. Notes We thank Steve Wise for his suggestions for improving this paper and Stefano Fusi and Nicolas Brunel for discussions. This study was supported by grants from The Israel Science Foundation of the Israel Academy of Arts and Sciences (#8009) and the German–Israel Science Foundation (GIF). Figure 1. View largeDownload slide Image set and task scheme. (a) Thirty ‘fractal’ images were used in the experiment, organized as 10 constant, non-overlapping triplets (columns A–J). Each image had a fixed ordinal position in its triplet (1st, 2nd and 3rd). (b) Series of events in a trial. Each trial began with a sequential presentation of three images of a given triplet. Each sample image was shown for 500 ms followed by a 1 s interval. Then the test stimulus was shown, consisting of the three sample images and a distractor at random screen positions. The distractor was randomly chosen from the other 27 images. The monkeys' task was to touch the three images that had been shown as samples in the order of the sample sequence, avoiding the distractor. Each correct touch was rewarded. Arrows in the ‘test’ image denote, for the reader, the correct touch order. Figure 1. View largeDownload slide Image set and task scheme. (a) Thirty ‘fractal’ images were used in the experiment, organized as 10 constant, non-overlapping triplets (columns A–J). Each image had a fixed ordinal position in its triplet (1st, 2nd and 3rd). (b) Series of events in a trial. Each trial began with a sequential presentation of three images of a given triplet. Each sample image was shown for 500 ms followed by a 1 s interval. Then the test stimulus was shown, consisting of the three sample images and a distractor at random screen positions. The distractor was randomly chosen from the other 27 images. The monkeys' task was to touch the three images that had been shown as samples in the order of the sample sequence, avoiding the distractor. Each correct touch was rewarded. Arrows in the ‘test’ image denote, for the reader, the correct touch order. Figure 2. View largeDownload slide Behavioral task variants for identifying complementary strategies. Schematic illustration of specific trial image set and distractor. A, B, . . . J correspond to the triplets of images; I, II, III represent the 1st, 2nd and 3rd ordinal number categories, respectively. White circles, images to be touched; black circle, distractor. Additional mnemonic strategies that can be used to complement categorization under each task variant are shown on the right. Arrows in the test stimulus denote the correct touch order. Schemes at top show order of presentation of images used. ‘Basic task’ trial: a triplet of fixed order images is presented, e.g. D1, D2, D3. In the case shown, besides identifying the images of the 1st category, the monkey also needs to discriminate between the correct image (D1) and the distractor (B1) for its first touch. This illustrates a choice between images that belong to the same ordinal category. Two supplementary routines are available: working memory& —& D1 just appeared as a sample stimulus while B1 did not; associations& —& since the same triplet was shown over and over again, associations between triplet members could be formed and used to avoid the distractor. ‘No samples’ trial variant: the three-sample image sequence is substituted by three successive blank rectangles and images are only shown simultaneously in the test phase, so that working memory is unavailable. However, the integrity of the triplets is maintained. The monkeys can therefore differentiate between the correct image and distractor belonging to the same category, on the basis of associations between successive images, formed during the learning stage. ‘Shuffled’ trial variant: associations between sequential items are precluded by presenting images from different triplets (e.g. J1, A2, C3) while maintaining the correct category order. The correct choice can be made using working memory of the samples. ‘No samples + shuffled’ trial: combining both manipulations. In this case, no strategy is available to discriminate between stimuli from the same category. The monkeys were rewarded for either choice. Figure 2. View largeDownload slide Behavioral task variants for identifying complementary strategies. Schematic illustration of specific trial image set and distractor. A, B, . . . J correspond to the triplets of images; I, II, III represent the 1st, 2nd and 3rd ordinal number categories, respectively. White circles, images to be touched; black circle, distractor. Additional mnemonic strategies that can be used to complement categorization under each task variant are shown on the right. Arrows in the test stimulus denote the correct touch order. Schemes at top show order of presentation of images used. ‘Basic task’ trial: a triplet of fixed order images is presented, e.g. D1, D2, D3. In the case shown, besides identifying the images of the 1st category, the monkey also needs to discriminate between the correct image (D1) and the distractor (B1) for its first touch. This illustrates a choice between images that belong to the same ordinal category. Two supplementary routines are available: working memory& —& D1 just appeared as a sample stimulus while B1 did not; associations& —& since the same triplet was shown over and over again, associations between triplet members could be formed and used to avoid the distractor. ‘No samples’ trial variant: the three-sample image sequence is substituted by three successive blank rectangles and images are only shown simultaneously in the test phase, so that working memory is unavailable. However, the integrity of the triplets is maintained. The monkeys can therefore differentiate between the correct image and distractor belonging to the same category, on the basis of associations between successive images, formed during the learning stage. ‘Shuffled’ trial variant: associations between sequential items are precluded by presenting images from different triplets (e.g. J1, A2, C3) while maintaining the correct category order. The correct choice can be made using working memory of the samples. ‘No samples + shuffled’ trial: combining both manipulations. In this case, no strategy is available to discriminate between stimuli from the same category. The monkeys were rewarded for either choice. Figure 3. View largeDownload slide Learning dynamics for the basic task, averaged across four monkeys. Touch performance is presented separately for cases where the distractor was from the same category as the correct image (‘same’ category, upper row) and cases in which the distractor was from a different category (‘different’ category, bottom row). Filled diamonds correspond to correct choices and filled triangles to distractor choices; the error bars denote 95% confidence interval. The number of trials per session decreases monotonically as a function of the touch number (1st, 2nd or 3rd) because an early wrong touch aborted the trial. Thus, generally, corresponding trials for the first touch occurred earlier in training than those of the 3rd touch. The dashed line in the ‘same’ category corresponds to touches of the correct category (i.e. sum of both correct and distractor touches). The average number of trials (across all monkeys) which were required to prefer the correct image over the distractor are shown for each condition by the arrow. Individual results are shown along the abscissa (asterisk, monkey J; open triangle, monkey S; open diamond, monkey G; open circle, monkey R). Each point reflects the individual's number of trials necessary for a preference of the correct image over the distractor (in statistically significant way, using a paired t-test, P < 0.01). In all touches, all monkeys learned to choose the correct image sooner when the distractor was from a ‘different’ category then when it was from the ‘same’ category (paired t-test, P < 0.00001). This shows that all monkeys initially relied on ordinal number categorization to perform the task. Only later did they learn to use other mnemonic strategies to select the correct item over the distractor from the same category. Figure 3. View largeDownload slide Learning dynamics for the basic task, averaged across four monkeys. Touch performance is presented separately for cases where the distractor was from the same category as the correct image (‘same’ category, upper row) and cases in which the distractor was from a different category (‘different’ category, bottom row). Filled diamonds correspond to correct choices and filled triangles to distractor choices; the error bars denote 95% confidence interval. The number of trials per session decreases monotonically as a function of the touch number (1st, 2nd or 3rd) because an early wrong touch aborted the trial. Thus, generally, corresponding trials for the first touch occurred earlier in training than those of the 3rd touch. The dashed line in the ‘same’ category corresponds to touches of the correct category (i.e. sum of both correct and distractor touches). The average number of trials (across all monkeys) which were required to prefer the correct image over the distractor are shown for each condition by the arrow. Individual results are shown along the abscissa (asterisk, monkey J; open triangle, monkey S; open diamond, monkey G; open circle, monkey R). Each point reflects the individual's number of trials necessary for a preference of the correct image over the distractor (in statistically significant way, using a paired t-test, P < 0.01). In all touches, all monkeys learned to choose the correct image sooner when the distractor was from a ‘different’ category then when it was from the ‘same’ category (paired t-test, P < 0.00001). This shows that all monkeys initially relied on ordinal number categorization to perform the task. Only later did they learn to use other mnemonic strategies to select the correct item over the distractor from the same category. Figure 4. View largeDownload slide Performance with different task variants. Touch choices of the correct (middle panel) and distractor images (lower panel) averaged across all three touches in two monkeys (J and R). A schematic diagram of an example trial is shown above the graphs for each task variant (see also Fig. 2). Touch proportions are shown separately for the ‘same’ and ‘different’ distractor conditions (striped and gray bars, respectively). Error bars denote 95% confidence interval for touch proportions. Note the almost flawless performance in the ‘different’ condition when ordinal position categorization suffices for correct response, compared to performance in the ‘same’ condition, which relies on the additional strategies available in each task version. Figure 4. View largeDownload slide Performance with different task variants. Touch choices of the correct (middle panel) and distractor images (lower panel) averaged across all three touches in two monkeys (J and R). A schematic diagram of an example trial is shown above the graphs for each task variant (see also Fig. 2). Touch proportions are shown separately for the ‘same’ and ‘different’ distractor conditions (striped and gray bars, respectively). Error bars denote 95% confidence interval for touch proportions. Note the almost flawless performance in the ‘different’ condition when ordinal position categorization suffices for correct response, compared to performance in the ‘same’ condition, which relies on the additional strategies available in each task version. Figure 5. View largeDownload slide Learning following category reassignment. Touch dynamics as a function of session number following category reassignment (shift 1, I–II–III → II–III–I). Images from the previous second category were then presented first, the former third category second, and images from the former first category were then shown last in each trial. Images were chosen at random from within categories precluding use of triplet associations. An example of a specific sequence is shown above. Only half of the images (five from each category) were used at this stage and no distractor was presented. Each session consisted of 100 trials. Note that category labels were changed but category membership remained intact: all images of a given category were assigned the same new label. Correct choices are depicted in hatched dark lines, erroneous choices according to the previous ordinal category, in black; other choices, in gray. The numbers I, II, III indicate the former category label in each touch. Note that the probabilities in the middle column are conditional on a correct response in the first, and those in the third column on those of the second. Initially, the monkeys tended to touch first the image belonging to the previous 1st category (left column, black curve), but with experience the image from the correct current category was chosen (hatched curve). Once a correct choice was made in the first touch, the monkeys seldom made errors in their second choice (middle column). This is presumably because their previous touch order was maintained (‘touch the previous 3rd after the 2nd’). On the other hand, the touch order was modified for the last touch (3rd → 1st). Indeed, monkey J initially had trouble in the last touch (right column), although by elimination there was only one possible untouched image left. Figure 5. View largeDownload slide Learning following category reassignment. Touch dynamics as a function of session number following category reassignment (shift 1, I–II–III → II–III–I). Images from the previous second category were then presented first, the former third category second, and images from the former first category were then shown last in each trial. Images were chosen at random from within categories precluding use of triplet associations. An example of a specific sequence is shown above. Only half of the images (five from each category) were used at this stage and no distractor was presented. Each session consisted of 100 trials. Note that category labels were changed but category membership remained intact: all images of a given category were assigned the same new label. Correct choices are depicted in hatched dark lines, erroneous choices according to the previous ordinal category, in black; other choices, in gray. The numbers I, II, III indicate the former category label in each touch. Note that the probabilities in the middle column are conditional on a correct response in the first, and those in the third column on those of the second. Initially, the monkeys tended to touch first the image belonging to the previous 1st category (left column, black curve), but with experience the image from the correct current category was chosen (hatched curve). Once a correct choice was made in the first touch, the monkeys seldom made errors in their second choice (middle column). This is presumably because their previous touch order was maintained (‘touch the previous 3rd after the 2nd’). On the other hand, the touch order was modified for the last touch (3rd → 1st). Indeed, monkey J initially had trouble in the last touch (right column), although by elimination there was only one possible untouched image left. Figure 6. View largeDownload slide Transfer test. Testing the degree of generalization in the learning dynamics. The upper panel shows a scheme of the presentation order in two sequential category reassignments (shift 1, II → III → I; shift 2, III → I → II). Images chosen in a specific trial are marked by white. No distractor was introduced in the test phase of the trials. The monkeys were first trained on only half the images (black frame) and then tested on the other half (hatched frame). The monkeys were trained on the second half of the images after achieving a criterion level of performance with the first half of the images. The 10 triplets were divided into two halves: one with triplets A, C, E, G, I and other with triplets B, D, F, H, J (see also Fig. 1). Monkey J learned the new labels with triplets A–C–E–G–I (first half), and then tested with triplets B–D–F–H–J (second half). Monkey S learned the new labels with triplets B–D–F–H–J (first half) and was tested with triplets A–C–E–G–I (second half). The percentage of correct touches of the entire triplet is plotted separately for the first (black curve) and second (hatched curve) halves of the image set as a function of the session number. Both monkeys learned the second half more rapidly than the first one after each shift. Error bars denote a 95% confidence interval for the touch proportions. The insets depict correct choices of the 1st touch during the first session at a finer temporal scale. Touch proportions are shown for each cycle of triplet repetition, averaged across all five images of the category. Smoothing is achieved by using a running average of the last five touches. Initial performance level was higher in the second half compared to the first, indicating a degree of generalization. Figure 6. View largeDownload slide Transfer test. Testing the degree of generalization in the learning dynamics. The upper panel shows a scheme of the presentation order in two sequential category reassignments (shift 1, II → III → I; shift 2, III → I → II). Images chosen in a specific trial are marked by white. No distractor was introduced in the test phase of the trials. The monkeys were first trained on only half the images (black frame) and then tested on the other half (hatched frame). The monkeys were trained on the second half of the images after achieving a criterion level of performance with the first half of the images. The 10 triplets were divided into two halves: one with triplets A, C, E, G, I and other with triplets B, D, F, H, J (see also Fig. 1). Monkey J learned the new labels with triplets A–C–E–G–I (first half), and then tested with triplets B–D–F–H–J (second half). Monkey S learned the new labels with triplets B–D–F–H–J (first half) and was tested with triplets A–C–E–G–I (second half). The percentage of correct touches of the entire triplet is plotted separately for the first (black curve) and second (hatched curve) halves of the image set as a function of the session number. Both monkeys learned the second half more rapidly than the first one after each shift. Error bars denote a 95% confidence interval for the touch proportions. The insets depict correct choices of the 1st touch during the first session at a finer temporal scale. Touch proportions are shown for each cycle of triplet repetition, averaged across all five images of the category. Smoothing is achieved by using a running average of the last five touches. Initial performance level was higher in the second half compared to the first, indicating a degree of generalization. Figure 7. View largeDownload slide A schematic framework for implementing ordinal number categorization. There are three basic types of neuronal populations organized in hierarchic levels and connected by two sets of plastic synapses. Each of the first populations is an abstract representation of one of the visual images (A1 . . . J3). It may serve as working memory if potentiation is sufficiently strong and may support (intra-triplet) context correlations. Activity of each of the second populations encodes an ordinal category label (I, II, III). These neurons are excited by a specific serial temporal position of either an image in the sample sequence or an anticipated touch during the test stimulus. Finally, the upper tier population reflects a decision about motor response order. Arrows denote synaptic connections that are potentiated during the learning stage. Dashed arrows correspond to connections established following a reassignment of categories (shift 1 in Experiment 3) even when only some of the images are trained. The framework predicts transfer of learning effects to untrained images of the same categories. Figure 7. View largeDownload slide A schematic framework for implementing ordinal number categorization. There are three basic types of neuronal populations organized in hierarchic levels and connected by two sets of plastic synapses. Each of the first populations is an abstract representation of one of the visual images (A1 . . . J3). It may serve as working memory if potentiation is sufficiently strong and may support (intra-triplet) context correlations. Activity of each of the second populations encodes an ordinal category label (I, II, III). These neurons are excited by a specific serial temporal position of either an image in the sample sequence or an anticipated touch during the test stimulus. Finally, the upper tier population reflects a decision about motor response order. Arrows denote synaptic connections that are potentiated during the learning stage. Dashed arrows correspond to connections established following a reassignment of categories (shift 1 in Experiment 3) even when only some of the images are trained. The framework predicts transfer of learning effects to untrained images of the same categories. Figure 8. View largeDownload slide Image affective valence effects during initial and final learning stages. The graph depicts the proportion of (erroneous) choices of a given distractor image (No. of distractor touches/No. of possible distractor touches& —& ordinate) as a function of the proportion of correct touches of the same image (No. of correct touches/No. of possible correct touches& —& abscissa). Possible distractor touches are defined as the number of trials in which an image is shown as a distractor times the number of touches in these trials. Similarly, the possible correct touches are defined as the number of trials in which the monkey had a chance to touch the image correctly. Thus, for example, for images belonging to the second ordinal category, only trials in which the correct first touch was made are taken into account. Each data point corresponds to the touch proportions of a different image (e.g. D2), averaged across two monkeys. Filled circles reflect choices of specific images from the 1st ordinal category; open circles are for images from the 2nd; gray circles are for images from the 3rd. Note that these proportions are gathered from different trials, since on each trial a given image was either a distractor or a target in one of the three touches. (a) Choices made in the first 15 sessions. (b) Choices in the later sessions. Initially, there is a striking correlation between the rate of choosing an image as a distractor and as a correct touch. With practice, this correlation was greatly reduced. Figure 8. View largeDownload slide Image affective valence effects during initial and final learning stages. The graph depicts the proportion of (erroneous) choices of a given distractor image (No. of distractor touches/No. of possible distractor touches& —& ordinate) as a function of the proportion of correct touches of the same image (No. of correct touches/No. of possible correct touches& —& abscissa). Possible distractor touches are defined as the number of trials in which an image is shown as a distractor times the number of touches in these trials. Similarly, the possible correct touches are defined as the number of trials in which the monkey had a chance to touch the image correctly. Thus, for example, for images belonging to the second ordinal category, only trials in which the correct first touch was made are taken into account. Each data point corresponds to the touch proportions of a different image (e.g. D2), averaged across two monkeys. Filled circles reflect choices of specific images from the 1st ordinal category; open circles are for images from the 2nd; gray circles are for images from the 3rd. Note that these proportions are gathered from different trials, since on each trial a given image was either a distractor or a target in one of the three touches. (a) Choices made in the first 15 sessions. (b) Choices in the later sessions. Initially, there is a striking correlation between the rate of choosing an image as a distractor and as a correct touch. With practice, this correlation was greatly reduced. References Amit D ( 1995) The Hebbian paradigm reintegrated: local reverberations as internal representations. Behav Brain Sci  18: 617–657. Google Scholar Amit D, Brunel N ( 1997) Global spontaneous activity and local structured (learned) delay activity in cortex. Cereb Cortex  7: 237–252. Google Scholar Amit D, Brunel N, Tsodyks M ( 1994) Correlations of cortical Hebbian reverberations: theory versus experiment. J Neurosci  14: 6435–6445. Google Scholar Barone P, Joseph, JP ( 1989) Prefrontal cortex and spatial sequencing in macaque monkey. Exp Brain Res  78: 447–464. Google Scholar Buckley MJ, Gaffan D ( 1998) Learning and transfer of object–reward associations and the role of the perirhinal cortex. Behav Neurosci  112: 15–23. Google Scholar Buckley MJ, Gaffan D ( 1998) Perirhinal cortex ablation impairs configural learning and paired-associate learning equally. Neuropsychologia  36: 535–546. Google Scholar Buckley MJ, Gaffan D ( 1998) Perirhinal cortex ablation impairs visual object identification. J Neurosci  18: 2268–2275. Google Scholar Buckley MJ, Gaffan D, Murray EA ( 1997) Functional double dissociation between two inferior temporal cortical areas: perirhinal cortex versus middle temporal gyrus. J Neurophysiol  77: 587–598. Google Scholar Buffalo B, Gaffan D, Murray EA ( 1994) A primacy effect in monkeys when list position is relevant. Q J Exp Psychol B  47: 353–369. Google Scholar Buffalo EA, Ramus SJ, Clark RE, Teng E, Squire LR, Zola SM ( 1999) Dissociation between the effects of damage to perirhinal cortex and area TE. Learn Mem  6: 572–599. Google Scholar Carpenter AF, Georgopoulos AP, Pellizzer G ( 1999) Motor cortical encoding of serial order in a context-recall task. Science  283: 1752–1757. Google Scholar Chen S, Swartz K, Terrace H ( 1997) Knowledge of the ordinal position of list items in rhesus monkeys. Psychol Sci  8: 80–86. Google Scholar Clower WT, Alexander GE ( 1998) Movement sequence-related activity reflecting numerical order of components in supplementary and pre-supplementary motor areas. J Neurophysiol  80: 1562–1566. Google Scholar D'Amato M, Colombo M ( 1988) Representation of serial order in monkeys (Cebula apella). J Exp Psychol Anim Behav Proc  14: 131–139. Google Scholar D'Amato M, Colombo M ( 1989) Serial learning with wild card items by monkeys (Cebula apella): implications for knowledge of ordinal position. J Comp Psychol  103: 252–263. Google Scholar D'Amato M, Colombo M ( 1990) The symbolic distance effect in monkeys (Cebula apella). Anim Learn Behav  18: 133–140. Google Scholar Ebbinghaus H (1964) Memory: a contribution to experimental psychology. New York: Dover. [Original work published 1885.] Google Scholar Ebenholtz S (1972) Serial learning and dimensional organization. In: The psychology of learning and motivation (Bower G, ed.), pp. 267–314. New York: Academic Press. Google Scholar Erickson CA, Desimone R ( 1999) Responses of macaque perirhinal neurons during and after visual stimulus association learning. J Neurosci  19: 10404–10416. Google Scholar Erickson CA, Jagadeesh B, Desimone R (2000) Learning and memory in the inferior temporal cortex of the macaque. In: The new cognitive neurosciences (Gazzaniga M, ed.), pp. 743–752. Cambridge, MA: MIT Press. Google Scholar Fuster JM ( 1973) Unit activity in prefrontal cortex during delayed-response performance: neuronal correlates of transient memory. J Neurophysiol  36: 61–78. Google Scholar Gower E ( 1992) Short-term memory for the temporal order of events in monkeys. Behav Brain Res  52: 99–103. Google Scholar Hebb D (1949) The organization of behavior: a neuropsychological theory. New York: Wiley. Google Scholar Kermadi I, Joseph JP ( 1995) Activity in the caudate nucleus of monkey during spatial sequencing. J Neurophysiol.  74: 911–933. Google Scholar McGonigle B, Chalmers M ( 1992) Monkeys are rational! Q J Exp Psychol  45: 189–228. Google Scholar Meunier M, Bachevalier J, Mishkin M, Murray E ( 1993) Effects on visual recognition of combined and separate ablations of the entorhinal and perirhinal cortex in rhesus monkeys. J Neurosci  13: 5418–5432. Google Scholar Miller EK, Erickson CA, Desimone R ( 1996) Neural mechanisms of visual working memory in prefrontal cortex of the macaque. J Neurosci  16: 5154–5167. Google Scholar Mishkin M ( 1982) A memory system in the monkey. Philos Trans R Soc Lond  298: 83–95. Google Scholar Miyashita Y ( 1988) Neuronal correlate of visual associative long-term memory in the primate temporal cortex. Nature  335: 817–820. Google Scholar Miyashita Y, Hayashi T ( 2000) Neural representation of visual objects: encoding and top-down Activation. Curr Opin Neurobiol  10: 187–194. Google Scholar Miyashita Y, Higuchi S, Sakai K, Masui N ( 1991) Generation of fractal patterns for probing the visual memory. Neurosci Res  12: 307–311. Google Scholar Murray EA, Gaffan D, Mishkin M ( 1993) Neural substrates of visual stimulus–stimulus association in Rhesus monkeys. J Neurosci  13: 4549–4561. Google Scholar Naya Y, Sakai K, Miyashita, Y ( 1996) Activity of primate infero-temporal neurons related to a sought target in pair-association task. Proc Natl Acad Sci USA  93: 2664–2669. Google Scholar Orlov T, Yakovlev V, Hochstein S, Zohary E ( 2000) Macaque monkeys categorize images by their ordinal number. Nature  404: 77–80. Google Scholar Parker A, Gaffan D ( 1998) Interaction of frontal and perirhinal cortices in visual object recognition memory in monkeys. Eur J Neurosci  10: 3044–3057. Google Scholar Picard N, Strick P ( 1997) Activation on the medial wall during remembered sequences of reaching movements in monkeys. J Neurophysiol  77: 2197–2201. Google Scholar Procyk E, Tanaka YL, Joseph JP ( 2000) Anterior cingulate activity during routine and non-routine sequential behaviors in macaques. Nat Neurosci  3: 502–508. Google Scholar Sakai K, Miyashita Y ( 1991) Neural organization for the long-term memory of paired associates. Nature  354: 152–155. Google Scholar Sands SF, Wright AA ( 1980) Serial probe recognition performance by a rhesus monkey and a human with  10- and 20-item lists. J Exp Psychol Anim Behav Proc 6: 386–396. Google Scholar Shima K, Tanji J ( 1998) Both supplementary and pre-supplementary motor areas are crucial for the temporal organization of multiple movements. J Neurophysiol  80: 3247–3260. Google Scholar Shima K, Sawamura J, Tanji J ( 2000) Neuronal activity in the superior parietal lobule reflecting numerical order of voluntary performed arm movements. Soc Neurosci Abstr  26: 1500. Google Scholar Suzuki WA, Miller EK, Desimone R ( 1997) Object and place memory in the macaque entorhinal cortex. J Neurophysiol  78: 1062–1081. Google Scholar Swartz K, Chen S, Terrace H ( 1991) Serial learning by rhesus monkeys: I. Acquisition and retention of multiple four-item lists. J Exp Psychol Anim Behav Proc  17: 396–410. Google Scholar Tang Y, Mishkin M, Aigner TG ( 1997) Effects of muscarinic blockade in perirhinal cortex during visual recognition. Proc Natl Acad Sci USA  94: 12667–12669. Google Scholar Tanji J, Shima K ( 1994) Role for supplementary motor area cells in planning several movements ahead. Nature  371: 413–416. Google Scholar Terrace H ( 1986) A non-verbal organism's knowledge of ordinal position in a serial learning task. J Exp Psychol Anim Behav Proc  41: 203–214. Google Scholar Terrace H ( 1987) Chunking by a pigeon in a serial learning task. Nature  325: 149–151. Google Scholar Terrace H ( 1993) The phylogeny and ontogeny of serial memory: list learning by pigeons and monkeys. Psychol Science  4: 162–169. Google Scholar Terrace H, Chen S, Newman A ( 1995) Serial learning with a wild card by pigeons: effect of list length. J Comp Psychol  109: 162–172. Google Scholar Thornton JA, Rothblat LA, Murray EA ( 1997) Rhinal cortex removal produces amnesia for preoperatively learned discrimination problems but fails to disrupt postoperative acquisition and retention in rhesus monkeys. J Neurosci  17: 8536–8549. Google Scholar Tomita H, Ohbayashi M, Nakahara K, Hasegawa I, Miyashita Y ( 1999) Top down signal from prefrontal cortex in executive control of memory retrieval. Nature  401: 699–703. Google Scholar Treichler F, van Tilburg D ( 1996) Concurrent conditional discrimination tests of transitive inference by macaque monkeys: list linking. J Exp Psychol Anim Behav Proc  22: 105–117. Google Scholar Yakovlev V, Fusi S, Berman E, Zohary E ( 1998) Inter-trial neuronal activity in inferior temporal cortex: a putative vehicle to generate long-term visual associations. Nat Neurosci  1: 310–317. Google Scholar © Oxford University Press
TI - Serial Memory Strategies in Macaque Monkeys: Behavioral and Theoretical Aspects
JF - Cerebral Cortex
DO - 10.1093/cercor/12.3.306
DA - 2002-03-01
UR - https://www.deepdyve.com/lp/oxford-university-press/serial-memory-strategies-in-macaque-monkeys-behavioral-and-theoretical-AY2a3XTfAm
SP - 306
EP - 317
VL - 12
IS - 3
DP - DeepDyve
ER -