Action Experience

Click here and press the right key for the next slide (or swipe left)

also ...

Press the left key to go backwards (or swipe right)

Press n to toggle whether notes are shown (or add '?notes' to the url before the #)

Press m or double tap to slide thumbnails (menu)

Press ? at any time to show the keyboard shortcuts

Step back and consider the problem more generally. The mind is made up of lots of different, loosely connected systems that work largely independently of each other. To a certain extent it’s fine for them to go their own way; and of course since they all get the same inputs (what with being parts of a single subject), there are limits on how separate the ways the go can be. Still, it’s often good for them to be aligned, at least eventually.

Experience is what enables there to be nonaccidental eventual alignment of largely independent cognitive systems. This is what experience is for.

Can we think along these lines in the case of action?

\textbf{What do we experience when someone acts?} One possibility is that we experience only bodily configurations, joint displacements and effects characteristic of particular actions. For the purposes of this talk, I will assume that this is wrong. Instead I will assume that we experience not only bodily configurations, joint displacements, sounds and the rest but also goal-directed actions.

What might it mean to experience action?

How are non-accidental matches possible?

action

vision

Could action experiences be like visual experiences? This wound imply that there are expeirences of action which stand to motor representations in something like the way that visual experiences stand to visual representations.

If this were right, how would it help with the interface problem?

Visual representations cause visual experiences, which provide reasons for beliefs.

Beliefs (and desires) influence orientation and attention, and thereby visual representations.

This seems unlikely for several reasons. First, vision involves a particular sensory modality. It would be quite radical to postulate a motor modality.

Second, the interface problem in the case of vision is, I suppose, primarily about how visual representations influence beliefs; you don’t want influence in the other direction, or at least not too much. By contrast, influence from intention to motor representation is essential; so vision seems likely to provide a poor for the case of action.

model 2: sensations (the feeling of familiarity)

model 3: object indexes

what is an object index? Formally, an object index is ‘a mental token that functions as a pointer to an object’ \citep[p.\ 11]{Leslie:1998zk}. If you imagine using your fingers to track moving objects, an object index is the mental counterpart of a finger \citep[p.~68]{pylyshyn:1989_role}.

The interesting thing about object indexes is that a system of object indexes (at least one, maybe more) appears to underpin cognitive processes which are not strictly perceptual but also do not involve beliefs or knowledge states. While I can’t fully explain the evidence for this claim here, I do want to mention one of the experimental tools that is used to investigate the existence of, and the principles underpinning, a system of object indexes which operates between perception and thought ...

[Object indexes are going to come in twice: once as a partial solution to the interface problem, then again as a model for a further conjecture about how it might be solved (the ‘action index’ conjecture).]

I was just saying that Experience is what enables there to be nonaccidental eventual alignment of largely independent cognitive systems. This is what experience is for.

object indexes

Object indexes are belief-independent. In this scenario, a patterned square disappears behind the barrier; later a plain black ring emerges. If you consider speed and direction only, these movements are consistent with there being just one object. But given the distinct shapes and textures of these things, it seems all but certain that there must be two objects. Yet in many cases these two objects will be assigned the same object index \citep{flombaum:2006_temporal,mitroff:2007_space}.

What do object indexes contribute to experience? Structure!

nonmodal. Not arrangement of surfaces in space and their boundaries (vision, touch). Instead we should think of them as giving \textbf{structure} to experience.

When a patterned square moves behind the occluder and a solid ring emerges on a spatio-temporal trajectory compatible with a single object, you can report the experience of a single object moving. But what is this experience? It’s not an experience of the surfaces, but nor is it an experience of something independent from the surfaces or of an entirely different element of experience. What you’re commenting on is something about the structure of your experience.

object indexes

belief-independent

structure experiences

subject to limited, indirect cognitive control through attention

There’s one major disanalogy with object indexes. Object indexes are about things which are actually present, whereas the motor representations we are interested in specify possible future outcomes.

This appears to be an objection because on the face of it, it seems that there could not be expeirence of future actions any more than we can experience future events.

So, you might object, the idea that motor representations structure experience is ok if you are merely observing someone act, but it will not help with performing actions.

Costantini et al, 2010 figure 1b

Part of the answer to the objection is that motor representations of outcomes may structure our experiences of objects. The existence of affordances suggests that this is at least possible.

So here is the idea: What is experienced is an object, not an action or an outcome. So the fact that the action lies in the future and the outcome has not yet occurred is no objection. But the motor representation of the outcome structures the experience of the object in some way. So the overall character of the experience of the mug differs when an action is represented motorically compared to when it does not.

And this difference is structural in the same sense that the difference object indexes make is structural. It’s about how elements of experience are organised rather than about any particular sensory modality.

‘Action index’ conjecture

Motor representations of outcomes structure
experiences, imaginings and (prospective) memories

in ways which provide opportunities for attention to actions directed to those outcomes.

Forming intentions concerning an outcome can influence attention to the action,

which can influence the persistence of a motor representation of the outcome.

A second objection: intentions and motor representations need to match in situations where you merely imagine acting or merely imaginary objects. In such situations there are no objects to experience.

Reply: motor representations structure experiences associated with imaginig things as they do experiences associated with actually perceiving things. To imagine acting on a mug (say), you need to imagine the mug.

A third objection: close your eyes, put yourself in a sensory deprivation chamber. Let your hand rest palm down on a table. Now intend to turn your hand palm up. Often enough, the intention will succeed. But by hypothesis there is nothing you experience (you are in a sensory deprivation chamber).

Two points in reply to this objection: first, we haven’t removed proprioception and other somasomatic senses. It may be that motor representations structure experiences of the body just as much as they structure experiences of mere objects.

But what if you remove somasomatic senses too? This is likely to impair action, but unlikely to make it impossible. Perhaps the ability to act in such situations depends on memory and imagination.

Predictions?

So far I’ve suggested that this conjecture (a) might contribute to solving the interface problem and (b) isn’t obviously wrong. But how could we tell whether it is right? What predictions does it generate?

1. Not all action-related changes in experience are merely changes in bodily configurations, movements and their sensory effects.

Prediction 1: it is possible to vary which action someone experiences while holding fixed her perceptual experiences of bodily configurations and joint displacements and their sensory effects.

2. Memory for objects should be influenced by their affordances.

Keyboard Shortcuts?

Action Experience

Keyboard Shortcuts`?`