CLIP Similarity Analysis

Overview

MagicEasel watches a participant's drawing, identifies objects in it, samples an object from the participant's surrounding environment, perturbs its description into an image-generation prompt, and returns an inspiration image. This analysis measures how semantically related — by CLIP cosine similarity — four stages of that process are to the participant's selected drawing object.

All four comparisons use the selected drawing-object text as their fixed reference. Only the item being compared changes.

The 728 blank-score rows are generations for which the drawing detector returned an empty list. They are retained in the CSV but contribute no CLIP values. The two conditions differ in their object source: Environmental draws from 15 shared objects; Personal draws from participant-provided photos.

Text-to-text and image-to-text scores have different numerical distributions. Cross-type comparisons (e.g., environment-text vs. generated-image) are not meaningful; same-type cross-condition comparisons are.

Pipeline

One generation follows this sequence. The selected drawing text (step 3) is the fixed CLIP reference for all four scores.

flowchart LR
    A["Drawing capture"] --> B["Object detection"]
    B --> C["Drawing target"]
    C --> D["Env object + photo"]
    D --> E["Image prompt"]
    E --> F["Generated image"]
    style C stroke:#B13254,stroke-width:2px
        

The text-embedding model used during step 4 to select the environment object is part of the original experiment. CLIP is a separate model applied afterward for this analysis.

CLIP model: openai/clip-vit-large-patch14 (L/14). Processor: openai/clip-vit-base-patch32 (B/32). Metric: raw cosine similarity between L2-normalized embeddings — not a percentage.

Dataset Coverage

The chart below shows, for each participant, how many of the retained rows contain all four CLIP scores (scored, darker) versus how many have blank scores due to empty drawing detections (blank, lighter). P4 is excluded entirely: its 86 archived generations cannot be reconstructed without a usable experiment log.

Condition-wide Means

Observed Personal means are higher for environment text, image prompt, and generated image. Environmental environment-photo mean is slightly higher. These are arithmetic means with no statistical test — no uncertainty analysis is included, and each participant contributes repeated rows.

Per-Participant Comparison

Each point is one participant (21 with both conditions). The dashed diagonal marks equal Environmental and Personal means. Points above the diagonal have a higher Personal mean; points below have a higher Environmental mean. Hover a point to see the participant ID and exact values. P3, P13 are Environmental-only; P4 is excluded.

Participant-Condition Means

Click any column header to sort. Rows with N scored < 5 are flagged — their means are averages over very few generations and should be interpreted with caution. P13 Personal has no scored rows because every retained generation had an empty detection list.

Data Quality

Pairing method breakdown

How drawing targets were associated with their generated output depends on whether requests were processed sequentially or concurrently. The vast majority were unique in their log position.

Specific exceptions

Sensitivity check for interleaved targets

A what-if comparison swapped the P16 and P20 target assignments and recomputed the 692-row Environmental means using the smaller openai/clip-vit-base-patch32 model:

Condition-wide differences are small because only 4 of 692 rows change. Within the four affected rows the largest mean difference is 0.0306; the P20 environment-text pair changes by 0.0443.

Reproduction

Install dependencies

The averaging script uses only Python's standard library and does not require CLIP once clip_scores.csv exists.

Recalculate row-level scores

The checked run produced 2,032 data rows and 58 columns. File size: 1,960,759 bytes. SHA-256: ee11a0be2bf4f4f09405e3222177523f613022da9025976d73f8e4f925fb5baf.

Recalculate participant-condition means

Prints the Markdown table in Section 5.3. Ignores all-blank rows; rejects rows with only some of the four scores; formats means to eight decimal places.

CLIP model and preprocessing

Shared Environmental Objects

The shared set contains 15 photos used across all Environmental sessions. Note: the misspelling envrionment (missing the second 'n') is part of the archived filenames and log values — code must preserve it when resolving paths.

Study Design Reference

The intended group contains the 24 participants listed below. P12, P22, earlier pilots, and _template directories exist in data2 but are outside this group. Background (Design / Non-Design) is descriptive metadata and does not affect score calculation.