Migrating a pre-existing DataJoint pipeline: MLAI and MouseAR¶

How the Mathis lab (MLAI) runs its MouseAR pipeline on the shared tables, with the code it actually uses:

Layer	Tables	Code
shared	Lab, Project, Subject, Session, Strain, Mouse	scene_pipelines
MLAI lab	MouseInfo, SessionInfo, ScoreSheet, …	base_schemas (Base-schemas)
MouseAR	Dataset, SessionDataset, …	SCENE_MouseAR

Run Getting started first. Then install the MLAI package (on Colab, the first code cell does this) (--no-deps keeps your installed scene_pipelines):

pip install --no-deps "git+https://github.com/SCENE-Collaboration/Base-schemas.git@dev"

In [ ]:

# On Google Colab only: a local MySQL server and the package. Elsewhere this cell does nothing.
import os
import sys

if "google.colab" in sys.modules:
    !apt-get -qq update > /dev/null && apt-get -qq install -y mysql-server > /dev/null
    !service mysql start
    !mysql -u root -e "ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'tutorial'"
    !pip install -q "scene-pipelines @ git+https://github.com/SCENE-Collaboration/SCENE-pipelines.git@v0.0.1"
    !pip install -q --no-deps "git+https://github.com/SCENE-Collaboration/Base-schemas.git@dev"
    !mkdir -p tutorial_utils && wget -q -O tutorial_utils/__init__.py https://raw.githubusercontent.com/SCENE-Collaboration/SCENE-pipelines/dev/examples/tutorials/tutorial_utils/__init__.py
    os.environ["DJ_PASS"] = "tutorial"

# On Google Colab only: a local MySQL server and the package. Elsewhere this cell does nothing. import os import sys if "google.colab" in sys.modules: !apt-get -qq update > /dev/null && apt-get -qq install -y mysql-server > /dev/null !service mysql start !mysql -u root -e "ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'tutorial'" !pip install -q "scene-pipelines @ git+https://github.com/SCENE-Collaboration/SCENE-pipelines.git@v0.0.1" !pip install -q --no-deps "git+https://github.com/SCENE-Collaboration/Base-schemas.git@dev" !mkdir -p tutorial_utils && wget -q -O tutorial_utils/__init__.py https://raw.githubusercontent.com/SCENE-Collaboration/SCENE-pipelines/dev/examples/tutorials/tutorial_utils/__init__.py os.environ["DJ_PASS"] = "tutorial"

1. Setup¶

An admin registers the lab and the project once (MouseAR: scripts/admin/register_scene_catalog.py).

In [ ]:

import json
import tempfile
from datetime import date, datetime
from pathlib import Path

import datajoint as dj
from tutorial_utils import connect

connect()

from base_schemas.ingestion import (
    MLAI_LAB,
    MLAI_LAB_KEY,
    ensure_mlai_experimenter,
    populate_base,
    register_mlai_mouse,
    scan_directory,
)
from base_schemas.queries import available_mice, mlai_mice
from base_schemas.schemas.mouse import ScoreSheet
from base_schemas.schemas.session import SessionInfo, SessionScoreSheet
from scene_pipelines.core import SCENE_REGISTRY, load_settings
from scene_pipelines.ingestion import subject_key
from scene_pipelines.ingestion.mouse import ensure_strain
from scene_pipelines.ingestion.provenance import DuplicatePolicy
from scene_pipelines.ingestion.scene import admin
from scene_pipelines.schemas.mouse.mouse import Mouse, Strain
from scene_pipelines.schemas.scene.session import Session
from scene_pipelines.schemas.scene.subject import Subject

SCENE_REGISTRY.activate_all()  # shared schemas and "mlai"

# SCENE_MouseAR: dj_pipeline/scene_project.py
MOUSEAR_PROJECT = {
    "project_name": "mousear",
    "project_title": "Augmented reality for freely moving mice",
}
MOUSEAR_PROJECT_KEY = {"project_name": MOUSEAR_PROJECT["project_name"]}

admin.ensure_lab(MLAI_LAB, if_exists=DuplicatePolicy.VERIFY)
admin.ensure_project(MOUSEAR_PROJECT, if_exists=DuplicatePolicy.VERIFY)

import json import tempfile from datetime import date, datetime from pathlib import Path import datajoint as dj from tutorial_utils import connect connect() from base_schemas.ingestion import ( MLAI_LAB, MLAI_LAB_KEY, ensure_mlai_experimenter, populate_base, register_mlai_mouse, scan_directory, ) from base_schemas.queries import available_mice, mlai_mice from base_schemas.schemas.mouse import ScoreSheet from base_schemas.schemas.session import SessionInfo, SessionScoreSheet from scene_pipelines.core import SCENE_REGISTRY, load_settings from scene_pipelines.ingestion import subject_key from scene_pipelines.ingestion.mouse import ensure_strain from scene_pipelines.ingestion.provenance import DuplicatePolicy from scene_pipelines.ingestion.scene import admin from scene_pipelines.schemas.mouse.mouse import Mouse, Strain from scene_pipelines.schemas.scene.session import Session from scene_pipelines.schemas.scene.subject import Subject SCENE_REGISTRY.activate_all() # shared schemas and "mlai" # SCENE_MouseAR: dj_pipeline/scene_project.py MOUSEAR_PROJECT = { "project_name": "mousear", "project_title": "Augmented reality for freely moving mice", } MOUSEAR_PROJECT_KEY = {"project_name": MOUSEAR_PROJECT["project_name"]} admin.ensure_lab(MLAI_LAB, if_exists=DuplicatePolicy.VERIFY) admin.ensure_project(MOUSEAR_PROJECT, if_exists=DuplicatePolicy.VERIFY)

2. Mice and experimenters¶

register_mlai_mouse splits one lab record: name, sex, birth date, strain and genotype go to the shared Subject and Mouse; the facility id stays in the lab's MouseInfo. For an experimenter, only the code is shared.

In [ ]:

c57 = ensure_strain({"strain_name": "C57BL/6J", "rrid": "RRID:IMSR_JAX:000664"})
ensure_mlai_experimenter("E01", "jdoe", mail="jane.doe@example.org")
register_mlai_mouse(
    "Tick",
    "F",
    mouse_id=1042,
    date_of_birth=date(2026, 1, 3),
    strain=c57,
    genotype="Cux2-CreERT2/wt; Ai148/wt",
)
mlai_mice("Tick")

c57 = ensure_strain({"strain_name": "C57BL/6J", "rrid": "RRID:IMSR_JAX:000664"}) ensure_mlai_experimenter("E01", "jdoe", mail="jane.doe@example.org") register_mlai_mouse( "Tick", "F", mouse_id=1042, date_of_birth=date(2026, 1, 3), strain=c57, genotype="Cux2-CreERT2/wt; Ai148/wt", ) mlai_mice("Tick")

3. Sessions from the transfer GUI¶

After each session, the GUI writes <mouse>_<date>_<attempt>.json. MouseAR's cron job (dj_pipeline/populate.py) calls populate_base on that folder: it validates each file, registers new sessions, and skips known ones.

In [ ]:

metadata = Path(tempfile.mkdtemp(prefix="mice_metadata_"))
score_sheet = {
    "license": "N/A",
    "body_condition": "BodyCondition3",
    "general_assay": "Assay5",
    "housing_assay": "Yes",
    "weight_percentage": "2.5",
}
for attempt in (1, 2):
    gui = {
        "mouse_name": "Tick",
        "doe": "2026-02-11",
        "attempt": attempt,
        "experimenter_name": "jdoe",
        "rig_id": 1,
        "task_name": "AR_visual_discrimination",
        **(score_sheet if attempt == 1 else {}),
    }
    (metadata / f"Tick_2026-02-11_{attempt}.json").write_text(json.dumps(gui))

print(scan_directory(metadata))  # the manifest; no database access
populate_base(metadata, project=MOUSEAR_PROJECT_KEY, suppress_errors=True)
assert populate_base(metadata, project=MOUSEAR_PROJECT_KEY) == []  # nothing new

(Session * SessionInfo & MLAI_LAB_KEY).proj("session_code", "attempt", "task_name")

metadata = Path(tempfile.mkdtemp(prefix="mice_metadata_")) score_sheet = { "license": "N/A", "body_condition": "BodyCondition3", "general_assay": "Assay5", "housing_assay": "Yes", "weight_percentage": "2.5", } for attempt in (1, 2): gui = { "mouse_name": "Tick", "doe": "2026-02-11", "attempt": attempt, "experimenter_name": "jdoe", "rig_id": 1, "task_name": "AR_visual_discrimination", **(score_sheet if attempt == 1 else {}), } (metadata / f"Tick_2026-02-11_{attempt}.json").write_text(json.dumps(gui)) print(scan_directory(metadata)) # the manifest; no database access populate_base(metadata, project=MOUSEAR_PROJECT_KEY, suppress_errors=True) assert populate_base(metadata, project=MOUSEAR_PROJECT_KEY) == [] # nothing new (Session * SessionInfo & MLAI_LAB_KEY).proj("session_code", "attempt", "task_name")

4. MouseAR: link datasets to sessions¶

MouseAR's tables hang off dataset_id, from the Unity file name. Only SessionDataset touches the shared layer: it points each dataset to its Session. Reduced copies of dj_pipeline/schemas/mousear_dataset.py.

The n-th dataset of a mouse on a day belongs to its n-th attempt. That link is decided by comparing other datasets and sessions, so SessionDataset is a Manual table filled by an explicit linking step, not a Computed one: a make() should read only its own key's upstream rows, or DataJoint cannot tell when to recompute it. Once the transfer GUI records the dataset with its session, the matching by position goes away.

In [ ]:

mousear = dj.Schema(load_settings().db_name("mousear_dataset"))


@mousear
class Dataset(dj.Manual):
    definition = """
    dataset_id: varchar(64)  # <mouse>_<yyyymmdd>_<hhmmss>, from the Unity file name
    ---
    session_date: datetime
    notes=NULL: varchar(256)
    """


@mousear
class SessionDataset(dj.Manual):
    definition = """
    -> Dataset
    ---
    -> Session
    """


def link_datasets(mouse_name: str, day: date) -> None:
    """Link a mouse's datasets of one day to its sessions of that day, by position."""
    same_mouse = (Dataset & f"dataset_id LIKE '{mouse_name}\\_%'").to_dicts()
    datasets = sorted(
        (d for d in same_mouse if d["session_date"].date() == day),
        key=lambda d: d["session_date"],
    )
    sessions = (
        Session * Session.Subject * SessionInfo
        & subject_key(mouse_name, lab=MLAI_LAB_KEY)
        & MOUSEAR_PROJECT_KEY
        & {"session_date": day}
    ).to_dicts(order_by="attempt")
    if len(datasets) != len(sessions):
        print(f"{mouse_name} on {day}: {len(datasets)} datasets, {len(sessions)} sessions; not linked")
        return
    SessionDataset.insert(
        [
            {"dataset_id": d["dataset_id"], **{name: s[name] for name in Session.primary_key}}
            for d, s in zip(datasets, sessions)
        ],
        skip_duplicates=True,
    )


Dataset.insert(
    [
        {"dataset_id": "Tick_20260211_091500", "session_date": datetime(2026, 2, 11, 9, 15)},
        {"dataset_id": "Tick_20260211_143000", "session_date": datetime(2026, 2, 11, 14, 30)},
    ],
    skip_duplicates=True,
)
link_datasets("Tick", date(2026, 2, 11))
SessionDataset * Session.proj("session_code")

mousear = dj.Schema(load_settings().db_name("mousear_dataset")) @mousear class Dataset(dj.Manual): definition = """ dataset_id: varchar(64) # <mouse>_<yyyymmdd>_<hhmmss>, from the Unity file name --- session_date: datetime notes=NULL: varchar(256) """ @mousear class SessionDataset(dj.Manual): definition = """ -> Dataset --- -> Session """ def link_datasets(mouse_name: str, day: date) -> None: """Link a mouse's datasets of one day to its sessions of that day, by position.""" same_mouse = (Dataset & f"dataset_id LIKE '{mouse_name}\\_%'").to_dicts() datasets = sorted( (d for d in same_mouse if d["session_date"].date() == day), key=lambda d: d["session_date"], ) sessions = ( Session * Session.Subject * SessionInfo & subject_key(mouse_name, lab=MLAI_LAB_KEY) & MOUSEAR_PROJECT_KEY & {"session_date": day} ).to_dicts(order_by="attempt") if len(datasets) != len(sessions): print(f"{mouse_name} on {day}: {len(datasets)} datasets, {len(sessions)} sessions; not linked") return SessionDataset.insert( [ {"dataset_id": d["dataset_id"], **{name: s[name] for name in Session.primary_key}} for d, s in zip(datasets, sessions) ], skip_duplicates=True, ) Dataset.insert( [ {"dataset_id": "Tick_20260211_091500", "session_date": datetime(2026, 2, 11, 9, 15)}, {"dataset_id": "Tick_20260211_143000", "session_date": datetime(2026, 2, 11, 14, 30)}, ], skip_duplicates=True, ) link_datasets("Tick", date(2026, 2, 11)) SessionDataset * Session.proj("session_code")

5. Queries across the layers¶

In [ ]:

# From a dataset up to the mouse's strain: MouseAR -> shared
(SessionDataset * Session.Subject * Subject * Mouse * Strain).proj(
    "subject_code", "strain_name", "rrid"
)

# From a dataset up to the mouse's strain: MouseAR -> shared (SessionDataset * Session.Subject * Subject * Mouse * Strain).proj( "subject_code", "strain_name", "rrid" )

In [ ]:

# The welfare check recorded with each dataset's session: MouseAR -> shared -> MLAI
(SessionDataset * SessionScoreSheet * ScoreSheet).proj("doc", "body_condition", "general_assay")

# The welfare check recorded with each dataset's session: MouseAR -> shared -> MLAI (SessionDataset * SessionScoreSheet * ScoreSheet).proj("doc", "body_condition", "general_assay")

In [ ]:

# MLAI mice available for experiments: not sacrificed, not breeders
available_mice().proj("subject_code", "mouse_id")

# MLAI mice available for experiments: not sacrificed, not breeders available_mice().proj("subject_code", "mouse_id")

6. Legacy tables¶

MLAI's old mice / exp tables are migrated with base_schemas.legacy.migrate_to_v001, through the same helpers. In MouseAR, with a JSON map of strains and experimenter codes:

python dj_pipeline/scripts/migrate_to_scene_pipelines.py migration_maps.json [--legacy HOST:PORT] [--skip-taken-ids]

To start over: from tutorial_utils import drop_tutorial_schemas; drop_tutorial_schemas(), then restart the kernel.