# Crucible: pull raw data, publish the analysis [Crucible](https://crucible.lbl.gov) is the Molecular Foundry's scientific data platform. Raw measurements live there as datasets, each with a permanent identifier (an MFID), its files, free-form scientific metadata, keywords, and links to samples, projects and other datasets. Its official Python client is [nano-crucible](https://molecularfoundrycrucible.github.io/nano-crucible/). xpsflow connects to it in four ways: 1. **Find** XPS datasets by MFID, by name, by project or by measurement type. 2. **Pull** a dataset's raw files into a local folder, keeping only the files xpsflow can read. 3. **Analyze** them with the standard pipeline. 4. **Publish** the analysis back as a new, private, *derived* dataset: the report, the results, a component table, a thumbnail and a compact metadata summary, linked to the raw dataset as its child and to the same samples and project. It also gives the Crucible client an **XPS parser**, so `crucible dataset create` extracts searchable metadata from VAMAS, Kratos and text exports at upload time. Nothing is ever written to Crucible unless you ask for it. Every publishing step is a dry run by default that prints exactly what would be uploaded. ## Install The integration is an optional extra, so the core install stays light: ```bash pip install "xpsflow[crucible]" # or: uv pip install -e ".[crucible]" ``` This installs `nano-crucible` (3.3 or later, below 5). Without the extra, every Crucible command and tool says which command to run instead of failing with a traceback. ## Credentials Crucible accounts sign in with ORCID. To get a key: 1. If you do not have an ORCID iD yet, register for one at [orcid.org/register](https://orcid.org/register). It is free and takes a few minutes. 2. Sign in to the [Crucible Hub](https://crucible.lbl.gov/hub/) with ORCID. The first sign-in creates your Crucible account. 3. Open [crucible.lbl.gov/api/v3/user_apikey](https://crucible.lbl.gov/api/v3/user_apikey) in the same browser. It shows your API key, signing you in with ORCID first if needed. Copy it straight into the next step. 4. Ask the Crucible team to add you to a project. A new account can read nothing and publish nothing until it belongs to one, and `crucible project list` shows which projects you have. xpsflow never asks for, stores or prints an API key. It builds the client the way nano-crucible does and lets it find the key: ```bash crucible config init # prompts for the key and writes nano-crucible's config file # or, for one shell or a CI job: export CRUCIBLE_API_KEY="..." ``` Environment variables take priority over the config file. Set `CRUCIBLE_API_URL` only to target a non-default deployment; the client ships with the production API URL. `crucible config path` shows where the config file lives on your system. Check the setup: ```bash xpsflow crucible status ``` ```text client: nano-crucible 3.3.0 api: https://crucible.lbl.gov/api/v3 reachable: yes (database ok) credentials: configured signed in as: ``` `status` exits with status 1 when the API is unreachable or the key is missing or rejected, so it can gate a script. ## Command line ```bash xpsflow crucible search telluride # XPS datasets whose name matches xpsflow crucible search --project my-project # every XPS dataset of a project xpsflow crucible search --mfid --json # one dataset, machine-readable xpsflow crucible search film --measurement "" # any measurement type xpsflow crucible pull --out runs/crucible # raw files into runs/crucible// xpsflow crucible analyze # pull, analyze, show what would be published xpsflow crucible analyze --publish # ... and publish it xpsflow crucible publish runs/latest --parent # dry run for an existing run xpsflow crucible publish runs/latest --parent --publish # publish it ``` `analyze` takes the same `--calibration` and `--background` choices as `xpsflow run`, and `--name` and `--project` to override the name and project the published dataset would inherit from its parent. Both `analyze` and `publish` also take: - `--sample ` (repeatable): Crucible samples to link, besides the parent's. Samples are linked by MFID and never created from names found in a file, as the Foundry's own uploaders do. - `--instrument-id` or `--instrument-mfid`: the registered XPS instrument. The default is the parent's instrument, and the dry run says so when there is none. - `--owner`: publish on a user's behalf with a service-account key, as instrument PCs do. Its working folder (default `runs/crucible//`) holds `raw//` with the downloaded files and Crucible's `record.json`, and `analysis/` with the usual run output. `publish` works on any folder written by `xpsflow run` or `crucible analyze`. Once a folder has been published it holds a `crucible_publication.json` receipt, and publishing it again is refused unless you pass `--again`, so a repeated command does not create a duplicate dataset. ### Which files are downloaded The reader registry decides. Files with an extension xpsflow reads (`.vms`, `.vamas`, `.kal`, `.txt`, `.csv`, `.dat`, `.xy`, `.tsv`, `.asc`) are downloaded; everything else (images, vendor binaries such as `.spe` or `.sle`, notebooks) is skipped and listed with the reason. Because a `.txt` or `.csv` can hold anything, each downloaded file is then opened with the readers, and a file that does not parse is deleted again and reported. A dataset with nothing readable is an error, not an empty analysis. Only files kept in Crucible's own storage can be downloaded; the client skips files that live elsewhere (a shared filesystem or a transfer service) and logs where they are. ## Uploading raw data with XPS metadata With the extra installed, the Crucible client discovers xpsflow's parser through the `crucible.parsers` entry point: ```bash crucible dataset parsers # lists "xps installed XPS" crucible dataset create -i sample.vms -t xps --project-id my-project ``` The parser reads the files with xpsflow's readers (no fitting) and attaches this metadata, with `XPS` and the elements as keywords, the measurement set to `XPS`, the data type to `XPS spectra`, and the dataset timestamp to the first acquisition time in the files. Pass the instrument, project and owner as for any `crucible dataset create`: | Field | Example | Meaning | |---|---|---| | `technique` | `["XPS"]` | technique recorded in the file | | `file_formats` | `["vamas"]` | reader that parsed the files | | `n_spectra`, `n_surveys`, `n_narrow_scans` | `6`, `1`, `5` | how many scans of each kind | | `regions` | `["survey", "C 1s", "Te 3d"]` | region labels | | `elements` | `["C", "O", "Te"]` | elements of the narrow scans | | `source`, `photon_energy_ev` | `["Al mono"]`, `[1486.6]` | X-ray source | | `pass_energy_ev` | `[20.0, 160.0]` | analyzer pass energies | | `lens_mode` | `["HSA_LENS_HYBRID"]` | lens mode, when recorded | | `energy_range_ev` | `{"C 1s": [280.0, 294.0]}` | binding-energy range of each region | | `acquired_from`, `acquired_to` | ISO timestamps | first and last acquisition | | `counting_time_s` | `1228.4` | total counting time (dwell × sweeps × points) | | `file_sample_ids`, `file_instrument`, `institution` | | the names the file records; Crucible samples and instruments are linked records, not these strings | | `parsed_with` | `"xpsflow 0.1.0"` | parser version | The legacy `crucible upload -t xps ...` uses the same parser and also attaches a plot of the first survey as the thumbnail. ## Python ```python from xpsflow.integrations import crucible as cr from xpsflow.pipeline import PipelineConfig # a list of {mfid, name, measurement, ...} cr.find_datasets(query="telluride") # raw files into runs/crucible// (files, skipped, record) pulled = cr.pull("", "runs/crucible") # pull, run the pipeline, plan the publication cfg = PipelineConfig(background="auto") outcome = cr.analyze("", "runs/crucible/", cfg) print(outcome.plan.describe()) # publish an existing run folder client = cr.get_client() plan = cr.plan_publication("runs/latest", parent="", client=client) cr.publish(plan, client=client) # dry run: sends nothing cr.publish(plan, client=client, publish=True) # creates the dataset and links ``` Every function takes an optional `client=` so you can pass a configured `CrucibleClient` (or a test double); without it the client is built from nano-crucible's own configuration. ## What is published One new dataset per analysis: | Field | Value | |---|---| | `dataset_name` | the parent's name plus `· XPS analysis` (or `--name`) | | `measurement` | `XPS analysis` | | `data_format` | `xpsflow` | | `data_type` | `XPS fit results` | | instrument | the parent's (or `--instrument-id` / `--instrument-mfid`) | | `timestamp` | when the analysis was published | | `owner` | `--owner`, when publishing for someone else | | `public` | `false` | | project, session | inherited from the parent (or `--project`) | | keywords | `XPS`, `xpsflow` and every element found | | files | `report.html`, `report.pdf` when the PDF extra is installed, `results.json`, `components.csv`, and, from the assistant or MCP, the [results package](results.md) (`-results.zip`) that CasaXPS and xpsflow reopen | | thumbnail | the fit figure of the most abundant element | Links: the new dataset becomes a **child of the raw dataset** with the relationship `is_derived_from`, and is linked to every **sample** the raw dataset is linked to, plus any given with `--sample`. A link that fails (for example for lack of permission on a sample) is reported and the rest still happen. `components.csv` has one row per fitted component: region, component, chemical state, assignment, centre, FWHM, area, its share of the region, its share of all atoms, and the region's audit verdict. The scientific metadata is a compact summary, small enough to search and facet on, with the full detail in `results.json`: ```json { "schema": "xpsflow.crucible.summary/1", "software": "xpsflow", "xpsflow_version": "0.1.0", "elements": ["C", "O", "Se", "Si", "Te"], "composition_atomic_percent": {"C": 19.06, "O": 41.06, "Te": 15.44, "Si": 19.39, "Se": 5.04}, "chemical_states_percent": { "demo/Te 3d": {"Te-M": 60.7, "Te(IV)": 39.3}, "demo/Si 2p": {"Si0": 41.8, "SiO2": 58.2} }, "calibration": {"mode": "adventitious_carbon", "applied": true, "shift_ev": -0.088, "reference_be": 284.8, "measured_be": 284.888, "reference_spectrum": "C 1s"}, "fit_audit": {"demo/C 1s": "review", "demo/Te 3d": "pass"}, "templates": {"demo/C 1s": "C1s", "demo/Te 3d": "Te3d"}, "n_spectra": 6, "n_warnings": 0, "provenance": { "results_sha256": "d0b133ea…", "inputs": [{"file": "demo.vms", "sha256": "54f7367d…"}], "references": 9 }, "source_dataset": "" } ``` `chemical_states_percent` gives each component's share *within its region*; the share of all atoms is in `components.csv`. `provenance.results_sha256` identifies the exact `results.json` uploaded next to it, and each input file is hashed when it is still on disk, so the derived dataset can be checked against the raw files it claims to come from. ## The assistant and MCP clients Three tools join the registry the assistant, the web workbench and the MCP server share: | Tool | Arguments | What it does | |---|---|---| | `crucible_search` | [query, mfid, project, measurement, limit] | Find datasets (read-only). | | `crucible_load` | mfid | Download a dataset into the session's data folder and load it like `load_data`. | | `crucible_publish` | [parent_mfid, name, confirm] | Return the publication plan; with `confirm=true`, publish it. | Downloads land inside the session's sandbox (the upload folder in the web workbench, the data directory of an MCP connection), and their total size is capped by `XPSFLOW_MAX_SESSION_UPLOAD_MB` (default 500). `crucible_publish` without `confirm` sends nothing; its description tells the model to set `confirm=true` only after you have seen the plan and asked to publish. The dataset loaded with `crucible_load` is the default parent; loading other files clears it. MCP clients see these tools marked as reaching an external service. These tools act with whatever Crucible credentials the machine running xpsflow has. On your own machine that is you. Do not configure Crucible credentials on a workbench or MCP server that other people can reach, or they will search, download and publish as you. The in-browser workbench has no Crucible client and does not offer these tools. ## Privacy - **Opt-in.** Searching and pulling only read. Publishing needs `--publish`, `publish=True` or `confirm=true`; without it you get a dry run. A dry run with `--parent` reads the parent record to show the inherited project and samples, and nothing else. - **Private by default.** Published datasets are created with `public=False`, the Crucible default. Making one public is a separate, deliberate step in Crucible (`crucible dataset set-public `). - **Only what is listed.** The dry run lists every file, keyword, link and metadata field that would be sent; nothing else from the run folder (figures, templates, the event log) is uploaded. - **No key handling.** The API key stays in nano-crucible's config file or your environment. ## Limits - Crucible may run its own ingestion on uploaded files. At the Foundry, instrument uploads get their file metadata from server-side ingestors (`crucible-ingestion`), and none handles XPS yet. The parser plugin here fills that gap from the client side. An XPS ingestor built on xpsflow's readers would put XPS on the standard instrument-PC path. xpsflow does not choose an ingestor or wait for it. - Linking needs permission on the parent dataset and on its samples; a refused link is reported in the result and does not undo the publication. - The parser extracts metadata only. It does not fit, calibrate or quantify; run `analyze` for that.