visual_search_experiment.qmd

---
title: "Visual Search Report"
author: "Dávid Kubek"
date: last-modified
date-format: long

bibliography: references.bib

format:
  html:
    toc: true
    theme: custom.scss
    code-fold: true
    code-line-numbers: true
  pdf:
    echo: false

execute: 
  cache: true
jupyter: python3
---

```{python}
#| code-summary: Imports and setup
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

import numpy as np
import seaborn as sns
import statsmodels.api as sm
import statsmodels.formula.api as smf
import pandas as pd
import matplotlib.pyplot as plt

from pathlib import Path

# classes for special types
from pandas.api.types import CategoricalDtype

# Apply the default theme
sns.set_theme(style="ticks")

# Set default precision for displaying numbers
pd.set_option("display.precision", 2)#| 
```


# Introduction

The visual search experiment is a psychology experiment that is used to study
how the human visual system processes and searches for specific stimuli in a
visual scene. The experiment typically involves presenting participants with a
visual display, such as a picture or a computer screen, that contains a number
of objects or stimuli. The participants are then asked to search for a specific
target stimulus, such as a pink line, among the other stimuli in the display.

During the experiment, the researcher may manipulate various factors, such as
the number of stimuli in the display, the similarity of the stimuli to the
target, or the location of the target within the display. These manipulations
allow the researcher to study how different factors affect the speed and
accuracy of the participant's search for the target stimulus.

One of the most widely used manipulation in visual search experiment is the set
size, which is the number of stimuli in the display. The experimenter increases
the set size and observe how it affects the reaction time of the participants. 

Another manipulation that is often used in visual search experiments is the
similarity of the stimuli to the target. This can include manipulating the
color, shape, or size of the stimuli to make them more or less similar to the
target. The experimenter can observe how these manipulations affect the speed
and accuracy of the participant's search for the target.

The visual search experiment is a powerful tool for studying the human visual
system because it allows researchers to study how the brain processes and
searches for specific stimuli in a visual scene. 

Overall, visual search experiment is a widely used method in cognitive
psychology to study how visual perception and attention work. It provides
researchers with a tool to investigate fundamental question about how we
perceive and process visual information in the world around us.

[@Wolfe:2008]

# Experiment Setup

We will study how the reaction time changes as set size increases under
different circumstances. Specifically, the effects of set size, homogeneity
(feature vs. conjunction) and target presence on reaction time will be examined.
A total of N=3 participants will be recruited for the study. We then fit a
linear regression model predicting the reaction time based on the set size,
search type and target presence. We will be interested in the coefficient in the
linear model corresponding to set size. We can think of the slope as giving us a
measure of how much does the reaction time increase, if we increase the set size
by 1.  This will be our measure of efficiency.

The experiment will consist of 300 trials, in which participants will be
presented with an image containing a set of lines with different orientations
(either ``/`` or ``\``). The participants will be instructed to search for a
pink line with an orientation of ``/`` and indicate whether it is present or
absent. The trials will vary in set size (10, 20, 30, or 40) and search type
(feature or conjunction). In _feature search_, the image will contain only lines
of a single color (lines with orientation ``\`` act as distractors), while in
_conjunction search_, the image will contain a mixture of blue and pink lines
(blue lines and pink lines with orientation ``\`` act as distractors).
Participants will be provided with feedback on the accuracy of their responses
and encouraged to respond as quickly as possible. The estimated duration of the
experiment is approximately 15-20 minutes.

The participants consisted of students in age range 20-25 years old. Both
genders were represented. The participants were seated in front of a 14-inch
monitor with 1920x1080 resolution and instructed to make themselves comfortable.
No training trials were conducted prior to the experiment. No participant needed
to take a break during the experiment.


```{python}
#| code-summary: Load data
 
data_path = Path('data/')

def load_file(filepath : Path) -> pd.DataFrame:
    participant_id = filepath.stem
    df = pd.read_csv(filepath)
    
    # Delete rows that do not contain response data
    df = df[df.trial_type == 'canvas-keyboard-response']
    
    df['participant'] = participant_id
    df = df.rename(
      {"setSize": "set_size"},
      axis=1
      )
    df = df[
      ['participant', 'present', 'set_size', 'conjuction','correct' , 'rt']
      ]
    df = df.astype(
      {
        "participant": int,
        "present": bool,
        'set_size': int, 
        'conjuction': bool,
        'correct': bool,
        'rt': int
      }
    )
    df = df.reset_index(drop=True)
    
    return df


df_all = pd.concat(
    load_file(file) for file in data_path.glob("*.csv")
  )\
  .reset_index(drop=True)
df_all = df_all[df_all['participant'] > 0]
```

```{python}
#| code-summary: Print sample of the data
#| tbl-cap: Sample of the input data.
#| warning: false
 
df_all.head()
```

```{python}
#| code-summary: Remove incorrect responses
df_correct = df_all[df_all.correct]
```

# Results

Firstly, in @tbl-accuracy we report the accuracy (percentage of errors for each
participant). Additionally @fig-accuracy we report the accuracy of the
participants on different search types as well as set sizes. We remove the error
trials for the subsequent analysis.

```{python}
#| code-summary: Report accuracy
#| label: tbl-accuracy
#| tbl-cap: Participant accuracy.
#| warning: false
 
hlp = df_all.groupby('participant')\
  .agg(accuracy=('correct', 'mean'))\
  .round(decimals=3)\
  .reset_index()
hlp = hlp.rename(
  {
    "participant": "Participant",
    "accuracy": "Accuracy",
  },
  axis='columns'
)

hlp
```


```{python}
#| code-summary: Report accuracy by homogeneity and participant
#| label: fig-accuracy
#| fig-cap: "Accuracy of of each participant (column) split by homogeneity (row)."
 
hlp = df_all.groupby(['participant','set_size', 'conjuction'])\
  .agg(
    Correct=('correct', 'mean'),
    Incorrect=('correct', lambda _: 1 - _.mean())
  )\
  .reset_index()
hlp = pd.melt(
  hlp,
  id_vars=['participant','set_size', 'conjuction'],
  var_name='is_correct', value_name='perc'
)
hlp['participant'] = hlp['participant'].astype(str)
hlp['set_size'] = hlp['set_size'].astype(str)
hlp['conjuction'] = hlp['conjuction'].map(
  lambda _: 'Conjunction' if _ else 'Feature'
  )
hlp = hlp.sort_values(['participant', 'set_size'])

g = sns.displot(
    data=hlp,
    x='set_size',
    hue='is_correct',
    multiple='stack',
    col='participant',
    row='conjuction',
    weights='perc',
    palette={'Correct': 'g', 'Incorrect': 'r'},
    hue_order=['Incorrect', 'Correct'],
    facet_kws=dict(margin_titles=True),
)
g.set_titles(col_template="Participant {col_name}", row_template="{row_name}")
g.set_ylabels("Accuracy")
g.set_xlabels("Set size")
g.legend.set_title("Answer")
g.fig.subplots_adjust(top=0.9)
g.fig.suptitle(
  "Accuracy for each participant split by set size and homogeneity."
  );
```

In @fig-rt we report the distribution of reaction times split by set size for
each participant.

```{python}
#| code-summary: Report reaction time split by set size
#| label: fig-rt
#| fig-cap: "Reaction times divided by set size."

hlp = df_correct[["participant", "set_size", "rt"]].copy()

# Treat set size as categorical data
hlp["set_size"] = hlp["set_size"].astype(str)

g = sns.catplot(
  data=hlp, 
  y="set_size", x="rt",
  hue='participant',
  kind='box',
  aspect=2,
  )
g.ax.xaxis.grid(True)
g.set_xlabels("Reaction time (ms)")
g.set_ylabels("Set size")
g.set(title="Reaction times of participants split by set size");
```

Next, we focus our attention on predicting the reaction times based on set size.
We are interested in how the performance changes if we change the homogeneity
(feature/conjunction) and target presence. In @tbl-results we report the set
size effect (search slope) for each case and participant with the 95% confidence
interval.

It is noteworthy that the slope for feature search is consistently lower
compared to conjunction search (when the target is present). This trend is also
observed for target absent trials in both feature and conjunction searches.
Lastly, the absence of the target leads to an increase in the search slope,
regardless of the search type.

These observations are also depicted in the @fig-slope.



```{python}
#| code-summary: Fit regression models

# Fit a linear regression model predicting the reaction time based on set size.
# Consider cases with different homogeneity and whether the target was present
# or not and report the slope of the coefficient for reaction time and 95%
# confidence interval. This time consider

participants = df_correct.participant.unique()
df_result = pd.DataFrame(
  columns=[
    "participant", "conjunction", "present", "set_size_slope", "ci_l", "ci_u", 
    "r2",
    ]
  )
for participant in participants:
    for is_conjunction in [True, False]:
        for target_present in [True, False]:
            hlp = df_correct[
                (df_correct.participant == participant) &
                (df_correct.conjuction == is_conjunction) &
                (df_correct.present == target_present)
            ]
            model = smf.ols('rt ~ set_size', data=hlp)
            result = model.fit()

            ci = result.conf_int(alpha=0.05).loc["set_size"]
            tmp = [
                participant,
                int(is_conjunction),
                int(target_present),
                result.params["set_size"],
                ci[0],
                ci[1],
                result.rsquared
            ]
            df_result = pd.concat(
              [df_result, pd.DataFrame([tmp], columns=df_result.columns)],
              ignore_index=True
              )

df_result = df_result.assign(
    yerr=lambda _: np.abs(_['set_size_slope'] - _['ci_l']),
)
```

```{python}
#| code-summary: Report search slope with confidence intervals
#| label: tbl-results
#| tbl-cap: Search slope with 95% confidence intervals.
#| warning: false
 
# Print pretty table
hlp = df_result[
    ['participant', 'conjunction', 'present', 'set_size_slope', 'ci_l','ci_u']
].copy()
hlp['conjunction'] = hlp['conjunction'].map(
  lambda _: 'Conjunction' if _ == 1 else 'Feature'
  )
hlp['present'] = hlp['present'].map(lambda _: 'Y' if _ == 1 else 'F')
hlp = hlp.rename(
    {
        "participant": "Participant",
        "conjunction": "Homogeneity",
        "present": "Target Present",
        "set_size_slope": "Search Slope",
        "ci_l": "95% CI Lower",
        "ci_u": "95% CI Upper",
    },
    axis='columns'
)
hlp.reset_index(drop=True)
```

```{python}
#| code-summary: Plot search slopes with confidence intervals
#| label: fig-slope
#| fig-cap: Search slope with 95% confidence intervals. The left figure contains the values for feature search, the right for conjunction search. Two values for each participant are shown. One for target present (circle) and one for target absent (square).
 
# Plot search slope for target present absent and feature or conjunction
from matplotlib.lines import Line2D

colors = ['C0', 'C1', 'C2']
markers = []
labels = []
for participant, color in enumerate(colors):
    for target_present, shape in [('present', 'o'), ('absent', 's')]:
        marker = Line2D(
            [], [], color=color, marker=shape, linestyle='none', markersize=5
        )
        markers.append(marker)
        labels.append(
          'Participant ' + str(participant + 1) + ', ' + target_present
          )
    
    
min_y = np.floor(df_result.ci_l.min()) - 1
max_y = np.ceil(df_result.ci_u.max()) + 1

fig, axes = plt.subplots(1, 2, figsize=(16, 5))

data = [
        ("Feature", df_result[df_result.conjunction == 0]),
        ("Conjunction", df_result[df_result.conjunction == 1])
]

delta = 0.1
for i, (title, df) in enumerate(data):
    for target_present in [True, False]:
        hlp = df[df.present == target_present]
        
        axes[i].scatter(
            hlp.participant + (-delta if target_present else delta),
            hlp.set_size_slope,
            color=colors,
        )

        for color, participant in zip(colors, hlp.participant):
            row = hlp[hlp.participant == participant]
            
            axes[i].errorbar(
                participant + (-delta if target_present else delta),
                row.set_size_slope,
                yerr=row.yerr,
                ls='none',
                marker='o' if target_present else 's',
                ecolor=color,
                c=color,
            )

    # Hide the right and top spines
    axes[i].spines[['right', 'top']].set_visible(False)
    axes[i].grid(which='major', axis='y', linestyle='-')
    axes[i].set_xticks(hlp.participant.astype(float))
    axes[i].set(title=title, xlabel='Participant', ylabel='Slope')
    axes[i].set_ylim(bottom=min_y, top=max_y)

fig.suptitle("Search slope with 95% confidence intervals")
fig.tight_layout()
fig.subplots_adjust(right=0.85)
fig.legend(
    markers,
    labels,
    loc=7,
);
```

# Conclusion

We now address the observed results.

First, we address the variations in slopes for different search types in the
presence of the target. The visual salience of the target in feature search
leads to it immediately capturing our attention as it drastically differs from
its surroundings. Conversely, the presence of distractors results in several
similar items attracting our attention, which necessitates the examination of
more items, thereby leading to an increase in reaction time, which agrees with
our observations.

Next we address the difference in slope for the target absent trials.  In target
present trials, it is generally necessary to examine roughly half of the set
size items until the target is found.  However, in the absence of the target,
the participant must spend more time examining the objects in search of a
target, which results in an increase in reaction time, which is consistent with
our observations.

# References