Getting started with SCENE pipelines¶
This tutorial registers the data of a small, made-up lab, the Demo lab, on the shared SCENE tables: two mice, a few sessions, and two tables of the lab's own. It ends with queries that run across all of them.
For the ideas behind the tables (layers, lab codes and ids), read the Introduction first.
Before you start (on Colab, the first code cell does all of this)
- Install the package from the repository root:
pip install -e . - Start the local MySQL database:
make initonce, thenmake db_up - Run this notebook from
examples/tutorials/, so thattutorial_utilscan be imported
Everything is written to databases named tutorial_*, never to your dev_ or production schemas.
You can run the notebook as often as you like: every step returns the same rows the second time.
# On Google Colab only: a local MySQL server and the package. Elsewhere this cell does nothing.
import os
import sys
if "google.colab" in sys.modules:
!apt-get -qq update > /dev/null && apt-get -qq install -y mysql-server > /dev/null
!service mysql start
!mysql -u root -e "ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'tutorial'"
!pip install -q "scene-pipelines @ git+https://github.com/SCENE-Collaboration/SCENE-pipelines.git@v0.0.1"
!mkdir -p tutorial_utils && wget -q -O tutorial_utils/__init__.py https://raw.githubusercontent.com/SCENE-Collaboration/SCENE-pipelines/dev/examples/tutorials/tutorial_utils/__init__.py
os.environ["DJ_PASS"] = "tutorial"
1. Connect and activate¶
connect() reads the connection from your shell or the repository's .env, and sets the schema prefix to tutorial_.
It also names this database: every database that SCENE writes to has a deployment id, here demo-tutorial.
from tutorial_utils import connect
connect()
The tables are plain Python classes: importing them needs no database.
Each shared schema registers itself in SCENE_REGISTRY, and activate_all() binds them all to the database
and creates the tables that do not exist yet.
from datetime import date
import datajoint as dj
from scene_pipelines.core import SCENE_REGISTRY, atomic
from scene_pipelines.ingestion import register_session, subject_key
from scene_pipelines.ingestion.mouse import ensure_strain, register_mouse
from scene_pipelines.ingestion.provenance import DuplicatePolicy
from scene_pipelines.ingestion.scene import admin
from scene_pipelines.schemas.mouse.mouse import Mouse, Strain
from scene_pipelines.schemas.scene.lab import Lab
from scene_pipelines.schemas.scene.project import Project
from scene_pipelines.schemas.scene.session import Experimenter, Session
from scene_pipelines.schemas.scene.subject import Subject
SCENE_REGISTRY.activate_all()
sorted(SCENE_REGISTRY.schemas)
2. Lab and project¶
Lab and Project are catalogs shared by the whole consortium. In production an administrator adds them once,
and your lab finds its rows already there. Here we add them ourselves.
DuplicatePolicy.VERIFY makes the call safe to repeat: the second time it checks that the stored row is the same
and returns its key.
lab = admin.ensure_lab(
{"lab_id": "demo", "lab_name": "Demo lab", "institution": "SCENE"},
if_exists=DuplicatePolicy.VERIFY,
)
project = admin.ensure_project(
{"project_name": "foraging", "project_title": "Foraging in an open arena"},
if_exists=DuplicatePolicy.VERIFY,
)
lab, project
Experimenters are known by a pseudonymous code within the lab. Names and e-mail addresses stay in the lab's own tables.
experimenter = {**lab, "experimenter_code": "E01"}
Experimenter.insert1(experimenter, skip_duplicates=True)
Experimenter & lab
3. Mice¶
Background strains are a shared vocabulary in official nomenclature: any lab adds the strains it uses.
register_mouse then writes the shared Subject and its Mouse row in one go.
The lab picks the code (mouse-001), which must be unique within the lab and must not identify anyone:
codes are shared with the whole consortium. The helper mints a global subject_id and returns it as the key.
c57 = ensure_strain({"strain_name": "C57BL/6J", "rrid": "RRID:IMSR_JAX:000664"})
MOUSE_1 = {"sex": "F", "date_of_birth": date(2026, 1, 15), "strain": c57}
mouse_1 = register_mouse("mouse-001", lab=lab, **MOUSE_1)
mouse_2 = register_mouse("mouse-002", "M", lab=lab, date_of_birth=date(2026, 1, 15), strain=c57)
Subject * Mouse & lab
Other kinds of subjects (humans, bats, agents) are registered with register_subject(code, kind, lab=lab).
Subject holds what every kind has in common; a kind-specific table such as Mouse adds the rest.
4. Sessions¶
A session has a lab code too, a date, and optionally a project, an experimenter and one or more subjects.
SESSION = {"lab": lab, "project": project, "experimenter": experimenter}
session_1 = register_session("mouse-001_20260302_1", date(2026, 3, 2), subjects=[mouse_1], **SESSION)
session_2 = register_session("mouse-002_20260302_1", date(2026, 3, 2), subjects=[mouse_2], **SESSION)
# Both mice in the arena together: one session, two subjects.
pair = register_session("pair_20260303_1", date(2026, 3, 3), subjects=[mouse_1, mouse_2], **SESSION)
(Session * Session.Subject * Subject & lab).proj("session_code", "session_date", "subject_code")
Each helper call is all-or-nothing. To make several calls all-or-nothing, wrap them in atomic:
if the session fails, the new mouse is not stored either.
with atomic(Session.connection):
mouse_3 = register_mouse("mouse-003", "U", lab=lab)
session_3 = register_session(
"mouse-003_20260304_1", date(2026, 3, 4), subjects=[mouse_3], **SESSION
)
5. Running it again¶
Registering the same code with the same content returns the same key, so an ingestion script can simply run again over all its data. The same code with different content raises an error instead of silently changing the stored row.
assert register_mouse("mouse-001", lab=lab, **MOUSE_1) == mouse_1
try:
register_mouse("mouse-001", lab=lab, **{**MOUSE_1, "sex": "M"})
except ValueError as err:
print(err)
This is the default policy, DuplicatePolicy.VERIFY. Pass if_exists= to choose another:
SKIP keeps the stored row, OVERWRITE replaces it, REJECT raises on any existing row.
To find a row by its lab code later, use subject_key (or session_key):
Mouse & subject_key("mouse-002", lab=lab)
6. The lab's own tables¶
Everything that is specific to the Demo lab goes into its own schema, registered on the same registry.
Lab tables hang off the shared rows (-> Session, -> Mouse): the shared row is the session's identity,
so there is no second session id to keep in sync.
The lab inserts into its own tables with plain DataJoint.
demo = SCENE_REGISTRY.get("demo") or SCENE_REGISTRY.make_schema("demo")
@demo
class Housing(dj.Manual):
definition = """
-> Mouse
---
cage: varchar(16)
"""
@demo
class SessionNotes(dj.Manual):
definition = """
-> Session
---
arena: varchar(32)
notes='': varchar(1000)
"""
SCENE_REGISTRY.activate_all() # binds the new "demo" schema
Housing.insert(
[{**mouse_1, "cage": "C-12"}, {**mouse_2, "cage": "C-12"}, {**mouse_3, "cage": "C-14"}],
skip_duplicates=True,
)
SessionNotes.insert1({**session_1, "arena": "open-field-1", "notes": "first session"}, skip_duplicates=True)
Session.proj("session_code", "session_date") * SessionNotes
7. Queries across the layers¶
All tables share their keys, so ordinary DataJoint joins work across the shared and the lab tables.
Every session of mouse-001, with the notes the lab added:
(
Session * Session.Subject & subject_key("mouse-001", lab=lab)
).proj("session_code", "session_date") * SessionNotes.proj("arena", "notes")
And the tables used in this tutorial, shared and lab's own:
diagram = dj.Diagram(demo)
for name in ("scene", "mouse"):
diagram += dj.Diagram(SCENE_REGISTRY.get(name))
diagram
Where next¶
- The Introduction explains the ideas behind these tables.
- An existing lab: MLAI and MouseAR: a real lab pipeline on the shared tables, with its automatic ingestion and legacy tables.
- The API reference documents every helper.
To start over, drop the tutorial databases and restart the kernel:
from tutorial_utils import drop_tutorial_schemas
drop_tutorial_schemas()