Crucible: pull raw data, publish the analysis

Crucible is the Molecular Foundry's scientific data platform. Raw measurements live there as datasets, each with a permanent identifier (an MFID), its files, free-form scientific metadata, keywords, and links to samples, projects and other datasets. Its official Python client is nano-crucible.

xpsflow connects to it in four ways:

1. Find XPS datasets by MFID, by name, by project or by measurement type.
2. Pull a dataset's raw files into a local folder, keeping only the files xpsflow can read.
3. Analyze them with the standard pipeline.
4. Publish the analysis back as a new, private, derived dataset: the report, the results, a component table, a thumbnail and a compact metadata summary, linked to the raw dataset as its child and to the same samples and project.

It also gives the Crucible client an XPS parser, so crucible dataset create extracts searchable metadata from VAMAS, Kratos and text exports at upload time.

Nothing is ever written to Crucible unless you ask for it. Every publishing step is a dry run by default that prints exactly what would be uploaded.

Install

The integration is an optional extra, so the core install stays light:

pip install "xpsflow[crucible]"        # or: uv pip install -e ".[crucible]"

This installs nano-crucible (3.3 or later, below 5). Without the extra, every Crucible command and tool says which command to run instead of failing with a traceback.

Credentials

Crucible accounts sign in with ORCID. To get a key:

1. If you do not have an ORCID iD yet, register for one at orcid.org/register. It is free and takes a few minutes.
2. Sign in to the Crucible Hub with ORCID. The first sign-in creates your Crucible account.
3. Open crucible.lbl.gov/api/v3/user_apikey in the same browser. It shows your API key, signing you in with ORCID first if needed. Copy it straight into the next step.
4. Ask the Crucible team to add you to a project. A new account can read nothing and publish nothing until it belongs to one, and crucible project list shows which projects you have.

xpsflow never asks for, stores or prints an API key. It builds the client the way nano-crucible does and lets it find the key:

crucible config init               # prompts for the key and writes nano-crucible's config file
# or, for one shell or a CI job:
export CRUCIBLE_API_KEY="..."

Environment variables take priority over the config file. Set CRUCIBLE_API_URL only to target a non-default deployment; the client ships with the production API URL. crucible config path shows where the config file lives on your system.

Check the setup:

xpsflow crucible status

client:       nano-crucible 3.3.0
api:          https://crucible.lbl.gov/api/v3
reachable:    yes (database ok)
credentials:  configured
signed in as: <your username>

status exits with status 1 when the API is unreachable or the key is missing or rejected, so it can gate a script.

Command line

xpsflow crucible search telluride                    # XPS datasets whose name matches
xpsflow crucible search --project my-project         # every XPS dataset of a project
xpsflow crucible search --mfid <MFID> --json         # one dataset, machine-readable
xpsflow crucible search film --measurement ""        # any measurement type

xpsflow crucible pull <MFID> --out runs/crucible     # raw files into runs/crucible/<MFID>/

xpsflow crucible analyze <MFID>                      # pull, analyze, show what would be published
xpsflow crucible analyze <MFID> --publish            # ... and publish it

xpsflow crucible publish runs/latest --parent <MFID>            # dry run for an existing run
xpsflow crucible publish runs/latest --parent <MFID> --publish  # publish it

analyze takes the same --calibration and --background choices as xpsflow run, and --name and --project to override the name and project the published dataset would inherit from its parent. Both analyze and publish also take:

• --sample <MFID> (repeatable): Crucible samples to link, besides the parent's. Samples are linked by MFID and never created from names found in a file, as the Foundry's own uploaders do.
• --instrument-id or --instrument-mfid: the registered XPS instrument. The default is the parent's instrument, and the dry run says so when there is none.
• --owner: publish on a user's behalf with a service-account key, as instrument PCs do. Its working folder (default runs/crucible/<MFID>/) holds raw/<MFID>/ with the downloaded files and Crucible's record.json, and analysis/ with the usual run output.

publish works on any folder written by xpsflow run or crucible analyze. Once a folder has been published it holds a crucible_publication.json receipt, and publishing it again is refused unless you pass --again, so a repeated command does not create a duplicate dataset.

Which files are downloaded

The reader registry decides. Files with an extension xpsflow reads (.vms, .vamas, .kal, .txt, .csv, .dat, .xy, .tsv, .asc) are downloaded; everything else (images, vendor binaries such as .spe or .sle, notebooks) is skipped and listed with the reason. Because a .txt or .csv can hold anything, each downloaded file is then opened with the readers, and a file that does not parse is deleted again and reported. A dataset with nothing readable is an error, not an empty analysis.

Only files kept in Crucible's own storage can be downloaded; the client skips files that live elsewhere (a shared filesystem or a transfer service) and logs where they are.

Uploading raw data with XPS metadata

With the extra installed, the Crucible client discovers xpsflow's parser through the crucible.parsers entry point:

crucible dataset parsers            # lists "xps  installed  XPS"
crucible dataset create -i sample.vms -t xps --project-id my-project

The parser reads the files with xpsflow's readers (no fitting) and attaches this metadata, with XPS and the elements as keywords, the measurement set to XPS, the data type to XPS spectra, and the dataset timestamp to the first acquisition time in the files. Pass the instrument, project and owner as for any crucible dataset create:

The legacy crucible upload -t xps ... uses the same parser and also attaches a plot of the first survey as the thumbnail.

Python

from xpsflow.integrations import crucible as cr
from xpsflow.pipeline import PipelineConfig

# a list of {mfid, name, measurement, ...}
cr.find_datasets(query="telluride")

# raw files into runs/crucible/<MFID>/ (files, skipped, record)
pulled = cr.pull("<MFID>", "runs/crucible")

# pull, run the pipeline, plan the publication
cfg = PipelineConfig(background="auto")
outcome = cr.analyze("<MFID>", "runs/crucible/<MFID>", cfg)
print(outcome.plan.describe())

# publish an existing run folder
client = cr.get_client()
plan = cr.plan_publication("runs/latest", parent="<MFID>", client=client)
cr.publish(plan, client=client)                # dry run: sends nothing
cr.publish(plan, client=client, publish=True)  # creates the dataset and links

Every function takes an optional client= so you can pass a configured CrucibleClient (or a test double); without it the client is built from nano-crucible's own configuration.

What is published

One new dataset per analysis:

Links: the new dataset becomes a child of the raw dataset with the relationship is_derived_from, and is linked to every sample the raw dataset is linked to, plus any given with --sample. A link that fails (for example for lack of permission on a sample) is reported and the rest still happen.

components.csv has one row per fitted component: region, component, chemical state, assignment, centre, FWHM, area, its share of the region, its share of all atoms, and the region's audit verdict.

The scientific metadata is a compact summary, small enough to search and facet on, with the full detail in results.json:

{
 "schema": "xpsflow.crucible.summary/1",
 "software": "xpsflow",
 "xpsflow_version": "0.1.0",
 "elements": ["C", "O", "Se", "Si", "Te"],
 "composition_atomic_percent": {"C": 19.06, "O": 41.06, "Te": 15.44, "Si": 19.39, "Se": 5.04},
 "chemical_states_percent": {
  "demo/Te 3d": {"Te-M": 60.7, "Te(IV)": 39.3},
  "demo/Si 2p": {"Si0": 41.8, "SiO2": 58.2}
 },
 "calibration": {"mode": "adventitious_carbon", "applied": true, "shift_ev": -0.088,
                 "reference_be": 284.8, "measured_be": 284.888, "reference_spectrum": "C 1s"},
 "fit_audit": {"demo/C 1s": "review", "demo/Te 3d": "pass"},
 "templates": {"demo/C 1s": "C1s", "demo/Te 3d": "Te3d"},
 "n_spectra": 6,
 "n_warnings": 0,
 "provenance": {
  "results_sha256": "d0b133ea…",
  "inputs": [{"file": "demo.vms", "sha256": "54f7367d…"}],
  "references": 9
 },
 "source_dataset": "<parent MFID>"
}

chemical_states_percent gives each component's share within its region; the share of all atoms is in components.csv. provenance.results_sha256 identifies the exact results.json uploaded next to it, and each input file is hashed when it is still on disk, so the derived dataset can be checked against the raw files it claims to come from.

The assistant and MCP clients

Three tools join the registry the assistant, the web workbench and the MCP server share:

Downloads land inside the session's sandbox (the upload folder in the web workbench, the data directory of an MCP connection), and their total size is capped by XPSFLOW_MAX_SESSION_UPLOAD_MB (default 500). crucible_publish without confirm sends nothing; its description tells the model to set confirm=true only after you have seen the plan and asked to publish. The dataset loaded with crucible_load is the default parent; loading other files clears it. MCP clients see these tools marked as reaching an external service.

These tools act with whatever Crucible credentials the machine running xpsflow has. On your own machine that is you. Do not configure Crucible credentials on a workbench or MCP server that other people can reach, or they will search, download and publish as you.

The in-browser workbench has no Crucible client and does not offer these tools.

Privacy

• Opt-in. Searching and pulling only read. Publishing needs --publish, publish=True or confirm=true; without it you get a dry run. A dry run with --parent reads the parent record to show the inherited project and samples, and nothing else.
• Private by default. Published datasets are created with public=False, the Crucible default. Making one public is a separate, deliberate step in Crucible (crucible dataset set-public <MFID>).
• Only what is listed. The dry run lists every file, keyword, link and metadata field that would be sent; nothing else from the run folder (figures, templates, the event log) is uploaded.
• No key handling. The API key stays in nano-crucible's config file or your environment.

Limits

• Crucible may run its own ingestion on uploaded files. At the Foundry, instrument uploads get their file metadata from server-side ingestors (crucible-ingestion), and none handles XPS yet. The parser plugin here fills that gap from the client side. An XPS ingestor built on xpsflow's readers would put XPS on the standard instrument-PC path. xpsflow does not choose an ingestor or wait for it.
• Linking needs permission on the parent dataset and on its samples; a refused link is reported in the result and does not undo the publication.
• The parser extracts metadata only. It does not fit, calibrate or quantify; run analyze for that.