CLI Reference¶
Overview¶
The bead CLI provides commands for all stages of the experimental pipeline, from resource creation through deployment and active learning. Commands are organized into logical groups corresponding to pipeline stages.
The CLI uses a hierarchical command structure:
All commands support --help for detailed usage information:
uv run bead --help # Show all command groups
uv run bead resources --help # Show all resource commands
uv run bead resources create-lexicon --help # Show specific command help
Global Options¶
These options apply to all bead commands:
| Option | Description |
|---|---|
--config-file PATH, -c PATH |
Path to configuration file (overrides profile defaults) |
--profile NAME, -p NAME |
Configuration profile: default, dev, prod, test |
--verbose, -v |
Enable verbose output with detailed logging |
--quiet, -q |
Suppress all output except errors |
--version |
Show bead version and exit |
--help |
Show help message and exit |
Examples:
# Use custom config file
uv run bead --config-file my-config.yaml config show
# Use development profile
uv run bead --profile dev resources list-lexicons
# Verbose output for debugging
uv run bead --verbose templates fill template.jsonl lexicon.jsonl filled.jsonl
# Quiet mode for scripts
uv run bead --quiet lists partition items.jsonl lists/ --n-lists 5
Command Groups¶
init¶
Initialize a new bead project with directory structure and default configuration.
Commands:
| Command | Description |
|---|---|
init [PROJECT_NAME] |
Create new project directory with scaffolding |
Options:
--profile NAME: Initialize with specific profile (default, dev, prod, test)--force: Overwrite existing directory
Examples:
# Create new project
uv run bead init my-experiment
# Initialize in current directory
uv run bead init
# Use development profile
uv run bead init my-experiment --profile dev
# Overwrite existing directory
uv run bead init my-experiment --force
Generated Structure:
my-experiment/
├── bead.yaml # Configuration file
├── .gitignore # Git ignores
├── lexicons/ # Lexical resources
├── templates/ # Template definitions
├── filled_templates/ # Generated filled templates
├── items/ # Generated items
├── lists/ # Experiment lists
├── experiments/ # Generated experiments
├── data/ # Collected data
└── models/ # Trained models
config¶
Configuration management commands.
Commands:
| Command | Description |
|---|---|
show |
Display current configuration with optional filtering |
validate |
Validate configuration file structure |
export |
Export merged configuration to YAML |
profiles |
List available configuration profiles |
merge |
Merge two configuration files |
create-active-learning |
Create active learning configuration template |
create-model |
Create model configuration template |
create-simulation |
Create simulation configuration template |
Examples:
# Show current configuration
uv run bead config show
# Show specific key
uv run bead config show --key paths.data_dir
# Show as JSON
uv run bead config show --format json
# Validate configuration file
uv run bead config validate
# Export with comments
uv run bead config export --output my-config.yaml --comments
# List available profiles
uv run bead config profiles
# Merge configurations
uv run bead config merge base.yaml overrides.yaml --output merged.yaml
# Create configuration templates
uv run bead config create-active-learning --output al_config.yaml
uv run bead config create-model --output model_config.yaml
resources (Stage 1)¶
Lexicon and template management commands.
Commands:
| Command | Description |
|---|---|
create-lexicon |
Create lexicon from CSV or JSON |
create-template |
Create template with slots |
generate-templates |
Generate templates from pattern |
generate-template-variants |
Generate systematic template variations |
list-lexicons |
List available lexicons |
list-templates |
List available templates |
validate-lexicon |
Validate lexicon file |
validate-template |
Validate template file |
import-verbnet |
Import from VerbNet |
import-unimorph |
Import from UniMorph |
import-propbank |
Import from PropBank |
import-framenet |
Import from FrameNet |
create-constraint |
Create constraint expression |
Common Examples:
# Create lexicon from CSV
uv run bead resources create-lexicon lexicon.jsonl --name verbs \
--from-csv verbs.csv --language-code eng
# Create template
uv run bead resources create-template template.jsonl --name transitive \
--template-string "{subject} {verb} {object}" \
--slot subject:true --slot verb:true --slot object:false
# Generate templates from pattern
uv run bead resources generate-templates template.jsonl \
--pattern "{subject} {verb} {object}" \
--name simple_transitive
# List lexicons in directory
uv run bead resources list-lexicons --directory lexicons/
# Validate lexicon
uv run bead resources validate-lexicon lexicon.jsonl
# Import from VerbNet
uv run bead resources import-verbnet verbs.jsonl --classes motion-51.1
# Create constraint
uv run bead resources create-constraint --expression "item['pos'] == 'verb'" \
--output constraints.jsonl
templates (Stage 2)¶
Template filling commands using various strategies.
Commands:
| Command | Description |
|---|---|
fill |
Fill templates with lexical items |
estimate-combinations |
Estimate total combinations |
list-filled |
List filled template files |
validate-filled |
Validate filled templates |
show-stats |
Show filling statistics |
filter-filled |
Filter filled templates by criteria |
merge-filled |
Merge multiple filled files |
export-csv |
Export to CSV format |
export-json |
Export to JSON array |
sample-combinations |
Sample with stratification |
Filling Strategies:
# Exhaustive filling (all combinations)
uv run bead templates fill template.jsonl lexicon.jsonl filled.jsonl \
--strategy exhaustive
# Random sampling
uv run bead templates fill template.jsonl lexicon.jsonl filled.jsonl \
--strategy random --max-combinations 100 --random-seed 42
# Stratified sampling
uv run bead templates fill template.jsonl lexicon.jsonl filled.jsonl \
--strategy stratified --max-combinations 100 \
--grouping-property pos --random-seed 42
Analysis and Processing:
# Estimate combinations before filling
uv run bead templates estimate-combinations template.jsonl lexicon.jsonl
# Show statistics
uv run bead templates show-stats filled.jsonl
# Filter by criteria
uv run bead templates filter-filled filled.jsonl filtered.jsonl \
--min-length 10 --template-name active
# Merge multiple files
uv run bead templates merge-filled file1.jsonl file2.jsonl merged.jsonl \
--deduplicate
# Export formats
uv run bead templates export-csv filled.jsonl filled.csv
uv run bead templates export-json filled.jsonl filled.json --pretty
items (Stage 3)¶
Item construction commands with task-type-specific creation functions.
Commands:
| Command | Description |
|---|---|
construct |
Construct items from filled templates |
create-forced-choice |
Create forced-choice item |
create-forced-choice-from-texts |
Create forced-choice items from text file |
create-nli |
Create NLI item |
create-likert-7 |
Create 7-point Likert item |
create-ordinal-scale-from-texts |
Create ordinal scale items from texts |
create-binary-from-texts |
Create binary judgment items from texts |
create-categorical |
Create categorical item |
create-multi-select-from-texts |
Create multi-select items from texts |
create-magnitude-from-texts |
Create magnitude estimation items |
create-free-text-from-texts |
Create free text response items |
create-simple-cloze |
Create simple cloze item |
list |
List item files |
validate |
Validate items file |
validate-for-task-type |
Validate items for specific task type |
show-stats |
Show item statistics |
infer-task-type |
Infer task type from item structure |
get-task-requirements |
Get requirements for task type |
Construction Examples:
# Basic item construction from templates
uv run bead items construct \
--item-template templates.jsonl \
--filled-templates filled.jsonl \
--output items.jsonl
# With constraints
uv run bead items construct \
--item-template templates.jsonl \
--filled-templates filled.jsonl \
--constraints constraints.jsonl \
--output items.jsonl
# With caching
uv run bead items construct \
--item-template templates.jsonl \
--filled-templates filled.jsonl \
--output items.jsonl \
--cache-dir .cache/models
# Create forced-choice items from texts
uv run bead items create-forced-choice-from-texts \
--texts-file sentences.txt \
--n-alternatives 2 \
--output items.jsonl
# Create Likert items
uv run bead items create-ordinal-scale-from-texts \
--texts-file sentences.txt \
--scale-min 1 --scale-max 7 \
--output items.jsonl
# Create NLI items
uv run bead items create-nli \
--premise "All dogs bark" \
--hypothesis "Some dogs bark" \
--output items.jsonl
Analysis:
# List items
uv run bead items list --directory items/
# Validate items
uv run bead items validate items.jsonl
# Validate for specific task type
uv run bead items validate-for-task-type items.jsonl --task-type forced_choice
# Show statistics
uv run bead items show-stats items.jsonl
# Infer task type
uv run bead items infer-task-type items.jsonl
# Get task type requirements
uv run bead items get-task-requirements forced_choice
lists (Stage 4)¶
List partitioning commands with constraint creation.
Commands:
| Command | Description |
|---|---|
partition |
Partition items into lists |
list |
List available experiment lists |
validate |
Validate list file |
show-stats |
Show list statistics |
create-uniqueness |
Create uniqueness constraint |
create-balance |
Create balance constraint |
create-quantile |
Create quantile constraint |
create-grouped-quantile |
Create grouped quantile constraint |
create-diversity |
Create diversity constraint |
create-size |
Create size constraint |
create-batch-coverage |
Create batch coverage constraint |
create-batch-balance |
Create batch balance constraint |
create-batch-diversity |
Create batch diversity constraint |
create-batch-min-occurrence |
Create batch min occurrence constraint |
Partitioning:
# Balanced partitioning
uv run bead lists partition items.jsonl lists/ --n-lists 5 \
--strategy balanced --random-seed 42
# Random partitioning
uv run bead lists partition items.jsonl lists/ --n-lists 5 \
--strategy random
# Stratified partitioning
uv run bead lists partition items.jsonl lists/ --n-lists 5 \
--strategy stratified
# Dry run preview
uv run bead lists partition items.jsonl lists/ --n-lists 5 \
--strategy balanced --dry-run
Creating Constraints:
# List constraints (per-list)
uv run bead lists create-uniqueness \
--property-expression "item.metadata.verb_lemma" \
-o constraints.jsonl
uv run bead lists create-balance \
--property-expression "item.metadata.condition" \
--tolerance 0.1 -o constraints.jsonl
# Batch constraints (across all lists)
uv run bead lists create-batch-coverage \
--property-expression "item.metadata.template_id" \
--target-values "0,1,2,3,4,5" \
--min-coverage 1.0 -o constraints.jsonl
uv run bead lists create-batch-balance \
--property-expression "item.metadata.verb_type" \
--target-distribution "transitive=0.5,intransitive=0.5" \
--tolerance 0.05 -o constraints.jsonl
Analysis:
# List available lists
uv run bead lists list --directory lists/
# Validate list
uv run bead lists validate lists/list_0.jsonl
# Show statistics
uv run bead lists show-stats lists/
deployment (Stage 5)¶
Experiment generation and deployment commands for jsPsych/JATOS.
Commands:
| Command | Description |
|---|---|
generate |
Generate jsPsych experiment |
export-jatos |
Export to JATOS .jzip format |
upload-jatos |
Upload to JATOS server |
validate |
Validate experiment directory |
ui |
UI customization subgroup |
trials |
Trial configuration subgroup |
UI Subgroup Commands:
ui customize- Apply UI customization to experimentui generate-css- Generate Material Design CSS
Trials Subgroup Commands:
trials configure-choice- Configure choice trial parameterstrials configure-rating- Configure rating scale parameterstrials configure-timing- Configure timing parameterstrials show-config- Display trial configuration
Generation:
# Basic generation with balanced distribution
uv run bead deployment generate lists/ items.jsonl experiment/ \
--experiment-type forced_choice \
--title "Acceptability Study" \
--distribution-strategy balanced
# Quota-based distribution
uv run bead deployment generate lists/ items.jsonl experiment/ \
--experiment-type forced_choice \
--distribution-strategy quota_based \
--distribution-config '{"participants_per_list": 25, "allow_overflow": false}'
# Stratified distribution
uv run bead deployment generate lists/ items.jsonl experiment/ \
--experiment-type forced_choice \
--distribution-strategy stratified \
--distribution-config '{"factors": ["condition", "verb_type"]}'
# Debug mode (always use same list)
uv run bead deployment generate lists/ items.jsonl experiment/ \
--experiment-type forced_choice \
--distribution-strategy balanced \
--debug-mode --debug-list-index 0
Distribution Strategies:
- random: Random list selection
- sequential: Round-robin assignment
- balanced: Assign to least-used list
- quota_based: Fixed quota per list
- latin_square: Counterbalancing with Bradley's algorithm
- weighted_random: Non-uniform probabilities
- stratified: Balance across metadata factors
- metadata_based: Filter and rank by metadata
UI Customization:
# Apply UI theme
uv run bead deployment ui customize experiment/ \
--theme dark --primary-color "#1976D2"
# Generate custom CSS
uv run bead deployment ui generate-css experiment/css/custom.css \
--theme dark --primary-color "#1976D2"
Trial Configuration:
# Configure rating scale
uv run bead deployment trials configure-rating \
--min-value 1 --max-value 7 \
--min-label "Completely unnatural" \
--max-label "Completely natural" \
--output rating_config.json
# Configure choice trials
uv run bead deployment trials configure-choice \
--button-html '<button class="choice-btn">%choice%</button>' \
--output choice_config.json
JATOS Integration:
# Export to .jzip
uv run bead deployment export-jatos experiment/ study.jzip \
--title "My Study" \
--description "Description text"
# Upload to JATOS server
uv run bead deployment upload-jatos study.jzip \
--jatos-url https://jatos.example.com \
--api-token YOUR_API_TOKEN
# Validate experiment
uv run bead deployment validate experiment/
models¶
GLMM model training commands for all 8 task types.
Commands:
| Command | Description |
|---|---|
train-model |
Train GLMM model for judgment prediction |
predict |
Make predictions with trained model |
predict-proba |
Predict class probabilities |
Mixed Effects Modes:
fixed: Fixed effects only (no participant variability)random_intercepts: Participant-specific biasesrandom_slopes: Participant-specific model parameters
Examples:
# Train forced choice model with fixed effects
uv run bead models train-model \
--task-type forced_choice \
--items items.jsonl \
--labels labels.jsonl \
--output-dir models/fc_model/
# Train with random intercepts
uv run bead models train-model \
--task-type ordinal_scale \
--items items.jsonl \
--labels labels.jsonl \
--participant-ids participant_ids.txt \
--mixed-effects-mode random_intercepts \
--output-dir models/os_model/
# Make predictions
uv run bead models predict \
--model-dir models/fc_model/ \
--items test_items.jsonl \
--output predictions.jsonl
# Predict probabilities
uv run bead models predict-proba \
--model-dir models/fc_model/ \
--items test_items.jsonl \
--output probabilities.jsonl
active-learning (Stage 6)¶
Active learning commands for convergence detection.
Note: This module is PARTIALLY IMPLEMENTED. Currently only convergence checking is available.
Commands:
| Command | Description |
|---|---|
check-convergence |
Check if model converged to human agreement |
Deferred Commands (awaiting implementation):
select-items- Requires model.predict_proba implementationrun- Requires data collection infrastructuremonitor-convergence- Requires checkpoint loading
Examples:
# Check convergence
uv run bead active-learning check-convergence \
--predictions predictions.jsonl \
--human-labels labels.jsonl \
--metric krippendorff_alpha \
--threshold 0.85
training¶
Data collection and model evaluation commands.
Commands:
| Command | Description |
|---|---|
collect-data |
Collect judgment data from JATOS |
show-data-stats |
Show statistics about collected data |
compute-agreement |
Compute inter-annotator agreement |
cross-validate |
Perform K-fold cross-validation |
evaluate |
Evaluate trained model on test set |
learning-curve |
Generate learning curve |
Examples:
# Collect data from JATOS
uv run bead training collect-data results.jsonl \
--jatos-url https://jatos.example.com \
--api-token TOKEN --study-id 123
# Show data statistics
uv run bead training show-data-stats results.jsonl
# Compute agreement
uv run bead training compute-agreement \
--annotations annotations.jsonl \
--metric krippendorff_alpha
# Cross-validation
uv run bead training cross-validate \
--items items.jsonl \
--labels labels.jsonl \
--model-config model_config.yaml \
--k-folds 5
# Evaluate model
uv run bead training evaluate \
--model-dir model_dir/ \
--test-items test_items.jsonl \
--test-labels test_labels.jsonl
# Generate learning curve
uv run bead training learning-curve \
--items items.jsonl \
--labels labels.jsonl \
--model-config model_config.yaml \
--output learning_curve.json
simulate¶
Simulation framework for testing active learning strategies.
Commands:
| Command | Description |
|---|---|
run |
Run simulation with configured annotators |
configure |
Create simulation configuration file |
analyze |
Analyze simulation results |
list-annotators |
List available annotator types |
list-noise-models |
List available noise models |
Annotator Types:
oracle- Perfect annotationsrandom- Random responseslm_score- Language model based scoringdistance- Distance-based similarity
Examples:
# List available annotators
uv run bead simulate list-annotators
# List noise models
uv run bead simulate list-noise-models
# Create configuration
uv run bead simulate configure \
--strategy lm_score \
--noise-type temperature \
--temperature 1.5 \
--output simulation_config.yaml
# Run simulation
uv run bead simulate run \
--items items.jsonl \
--templates templates.jsonl \
--annotator lm_score \
--n-annotators 5 \
--output results.jsonl
# Analyze results
uv run bead simulate analyze results.jsonl \
--metrics agreement accuracy convergence \
--output analysis.json
workflow¶
High-level workflow commands for complete pipeline execution.
Commands:
| Command | Description |
|---|---|
run |
Run complete pipeline workflow |
init |
Initialize new project from template |
status |
Show current workflow status |
resume |
Resume interrupted workflow |
rollback |
Rollback to previous stage |
list-templates |
List available workflow templates |
Pipeline Stages:
- resources - Create lexicons and templates
- templates - Fill templates with lexicon items
- items - Construct experimental items
- lists - Partition items into experiment lists
- deployment - Generate jsPsych experiments
- training - Train models with active learning (optional)
Examples:
# Run all stages
uv run bead workflow run --config bead.yaml
# Run specific stages
uv run bead workflow run --stages resources,templates,items
# Start from items stage
uv run bead workflow run --from-stage items
# Dry run to preview
uv run bead workflow run --dry-run
# Initialize from template
uv run bead workflow init acceptability-study
# Check status
uv run bead workflow status
# Resume interrupted workflow
uv run bead workflow resume
# Rollback to deployment stage
uv run bead workflow rollback deployment
# List available templates
uv run bead workflow list-templates
completion¶
Shell completion setup.
Commands:
| Command | Description |
|---|---|
bash |
Generate bash completion script |
zsh |
Generate zsh completion script |
fish |
Generate fish completion script |
install |
Auto-detect shell and install completion |
Examples:
# Generate bash completion
uv run bead completion bash > ~/.bash_completion.d/bead
# Generate zsh completion
uv run bead completion zsh > ~/.zsh/completion/_bead
# Generate fish completion
uv run bead completion fish > ~/.config/fish/completions/bead.fish
# Auto-install for current shell
uv run bead completion install
shell¶
Interactive shell commands.
Commands:
| Command | Description |
|---|---|
repl |
Start interactive REPL shell |
Examples:
Common Workflows¶
Complete Pipeline (Stages 1-5)¶
Build experiment from scratch:
# Stage 1: Create resources
uv run bead resources create-lexicon lexicons/verbs.jsonl --name verbs \
--from-csv data/verbs.csv --language-code eng
uv run bead resources create-template templates/transitive.jsonl \
--name transitive \
--template-string "{subject} {verb} {object}"
# Stage 2: Fill templates
uv run bead templates fill templates/transitive.jsonl \
lexicons/verbs.jsonl \
filled_templates/filled.jsonl \
--strategy exhaustive
# Stage 3: Construct items
uv run bead items construct \
--item-template item_templates/templates.jsonl \
--filled-templates filled_templates/filled.jsonl \
--output items/items.jsonl
# Stage 4: Partition into lists
uv run bead lists partition items/items.jsonl lists/ \
--n-lists 5 --strategy balanced --random-seed 42
# Stage 5: Generate experiment
uv run bead deployment generate lists/ items/items.jsonl experiment/ \
--experiment-type forced_choice \
--title "Acceptability Study" \
--distribution-strategy balanced
Active Learning Workflow¶
Train model with convergence detection:
# Collect initial data
uv run bead training collect-data data/initial_labels.jsonl \
--jatos-url https://jatos.example.com \
--api-token TOKEN --study-id 123
# Train initial model
uv run bead models train-model \
--task-type forced_choice \
--items items/items.jsonl \
--labels data/initial_labels.jsonl \
--output-dir models/model_v1/
# Check convergence
uv run bead active-learning check-convergence \
--predictions models/model_v1/predictions.jsonl \
--human-labels data/initial_labels.jsonl \
--metric krippendorff_alpha \
--threshold 0.8
# Generate learning curve
uv run bead training learning-curve \
--items items/items.jsonl \
--labels data/all_labels.jsonl \
--model-config model_config.yaml \
--output learning_curve.json
Simulation Testing¶
Test strategies before deployment:
# List available annotators
uv run bead simulate list-annotators
# Create simulation configuration
uv run bead simulate configure \
--strategy lm_score \
--noise-type temperature \
--temperature 0.7 \
--output sims/config.yaml
# Run simulation
uv run bead simulate run \
--items items/items.jsonl \
--templates templates/templates.jsonl \
--annotator lm_score \
--n-annotators 5 \
--output sims/results.jsonl
# Analyze results
uv run bead simulate analyze sims/results.jsonl \
--metrics agreement accuracy convergence \
--output sims/evaluation.json
JATOS Deployment¶
Deploy to JATOS server:
# Generate experiment
uv run bead deployment generate lists/ items/items.jsonl experiment/ \
--experiment-type forced_choice \
--title "Acceptability Study" \
--description "Judge sentence acceptability" \
--distribution-strategy balanced
# Customize UI
uv run bead deployment ui customize experiment/ \
--theme dark --primary-color "#1976D2"
# Validate before export
uv run bead deployment validate experiment/
# Export to .jzip
uv run bead deployment export-jatos experiment/ acceptability_study.jzip \
--title "Acceptability Study" \
--description "Version 1.0"
# Upload to JATOS
uv run bead deployment upload-jatos acceptability_study.jzip \
--jatos-url https://jatos.example.com \
--api-token YOUR_API_TOKEN
Configuration-Driven Workflow¶
Use YAML configuration for entire pipeline:
# Create project with config
uv run bead init my-experiment --profile prod
# Edit config file (bead.yaml)
# Specify all pipeline parameters
# Validate configuration
uv run bead config validate
# Run entire pipeline from config
uv run bead workflow run --config bead.yaml
# Check pipeline status
uv run bead workflow status
# Resume if interrupted
uv run bead workflow resume
Tips and Best Practices¶
Configuration Management¶
Use profiles for different environments:
# Development: verbose output, small samples
uv run bead --profile dev templates fill ... --max-combinations 10
# Production: optimized, full datasets
uv run bead --profile prod workflow run --config bead.yaml
# Show configuration with specific key
uv run bead config show --key paths.data_dir
# Export with comments for documentation
uv run bead config export --output full-config.yaml --comments
Error Handling¶
Use validation commands before expensive operations:
# Validate inputs before filling
uv run bead resources validate-lexicon lexicon.jsonl
uv run bead resources validate-template template.jsonl
# Estimate before exhaustive filling
uv run bead templates estimate-combinations template.jsonl lexicon.jsonl
# Dry-run before partitioning
uv run bead lists partition items.jsonl lists/ --n-lists 5 --dry-run
# Validate before deployment
uv run bead deployment validate experiment/
Performance Optimization¶
# Use caching for repeated operations
uv run bead items construct ... --cache-dir .cache/models
# Use random sampling for large spaces
uv run bead templates fill ... --strategy random --max-combinations 1000
# Preview with dry-run before expensive operations
uv run bead workflow run --dry-run
Reproducibility¶
Always set random seeds:
# Template filling
uv run bead templates fill ... --strategy random --random-seed 42
# List partitioning
uv run bead lists partition ... --random-seed 42
# Model training
uv run bead models train-model ... --random-seed 42
Debugging¶
Use verbose mode and dry-runs:
# Verbose output
uv run bead --verbose templates fill ...
# Dry-run preview
uv run bead lists partition ... --dry-run
uv run bead workflow run --dry-run
# Debug mode for deployment
uv run bead deployment generate ... --debug-mode --debug-list-index 0
# Show configuration for debugging
uv run bead config show --format json --no-redact
File Organization¶
Follow recommended directory structure:
project/
├── bead.yaml # Configuration
├── lexicons/ # Stage 1 outputs
├── templates/ # Stage 1 outputs
├── filled_templates/ # Stage 2 outputs
├── items/ # Stage 3 outputs
├── lists/ # Stage 4 outputs
├── experiments/ # Stage 5 outputs
├── data/ # Collected responses
├── models/ # Trained models (Stage 6)
└── .cache/ # Model output cache