Run plaknit inside Singularity/Apptainer¶

This walkthrough reflects the latest working recipe for launching the plaknit CLI inside an OTB-enabled Singularity/Apptainer container on a shared HPC cluster. The emphasis is on keeping everything writable inside your project space while the container remains read-only.

Requirements¶

Writable project or scratch directory (for example /blue/$USER/..., /scratch/$USER, or /project/...).
Singularity/Apptainer module available on the cluster.
Container image that already includes Orfeo ToolBox (OTB), GDAL, and Python 3.8 or newer.

Same CLI everywhere

plaknit runs the same inside or outside the container. Once the venv is active you can reuse the commands from Usage verbatim.

1. Directory layout¶

Set your PROJECT_DIR and SIF paths and create the file structure:

export PROJECT_DIR=/path/to/your/project
export STRIPS=$PROJECT_DIR/data/strips
export UDMS=$PROJECT_DIR/data/udms
export OUTDIR=$PROJECT_DIR/output
export VENVBASE=$PROJECT_DIR/venvs
export PIPCACHE=$PROJECT_DIR/cache/pip
export BOOT=$PROJECT_DIR/bootstrap
export SIF=otb.sif

mkdir -p "$STRIPS" "$UDMS" "$OUTDIR" "$VENVBASE" "$PIPCACHE" "$BOOT"

Note

Keep these assets under quota-friendly paths instead of $HOME, especially on clusters that enforce small home-directory limits.

2. Download get-pip (Python 3.8 example)¶

Many OTB images ship Python without pip. Grab the version-specific bootstrap script so you can install pip in a writable prefix:

wget -fsSLo "$BOOT/get-pip.py" https://bootstrap.pypa.io/pip/3.8/get-pip.py

Use the matching URL for Python 3.9+ images.

3. Install pip inside the container¶

singularity exec \
  --bind "$BOOT":/bootstrap \
  --bind "$VENVBASE":/venvs \
  "$SIF" bash -lc '
    set -euo pipefail
    mkdir -p /venvs/piproot
    python3 /bootstrap/get-pip.py --prefix /venvs/piproot

    PYVER=$(python3 -c '\''import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")'\'')
    export PYTHONPATH=/venvs/piproot/lib/python${PYVER}/site-packages
    export PATH=/venvs/piproot/bin:$PATH

    pip --version
  '

This keeps every Python artifact inside $PROJECT_DIR/venvs.

4. Create the persistent plaknit virtualenv¶

singularity exec \
  --bind "$VENVBASE":/venvs \
  --bind "$PIPCACHE":/pipcache \
  "$SIF" bash -lc '
    set -euo pipefail
    PYVER=$(python3 -c '\''import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")'\'')
    export PYTHONPATH=/venvs/piproot/lib/python${PYVER}/site-packages
    export PATH=/venvs/piproot/bin:$PATH

    pip install --cache-dir /pipcache virtualenv
    virtualenv --python=python3 /venvs/plaknit
    source /venvs/plaknit/bin/activate
    pip install --cache-dir /pipcache plaknit

    echo "[verify] python -> $(which python)"
    echo "[verify] plaknit -> $(which plaknit)"
    plaknit --version || plaknit --help
  '

The venv now lives at $PROJECT_DIR/venvs/plaknit and can be reused across jobs.

4a. Install classify dependencies (GeoPandas/Fiona)¶

plaknit classify relies on GeoPandas + Fiona. Install them inside the container venv (installing on the host will not affect the container):

singularity exec \
  --bind "$VENVBASE":/venvs \
  --bind "$PIPCACHE":/pipcache \
  "$SIF" bash -lc '
    set -euo pipefail
    source /venvs/plaknit/bin/activate
    pip install --cache-dir /pipcache "geopandas>=0.13" "fiona>=1.9"
    python -c "import fiona, geopandas; print(\"fiona\", fiona.__version__, \"geopandas\", geopandas.__version__)"
  '

If you see AttributeError: module 'fiona' has no attribute 'path', you are running with an incompatible Fiona/GeoPandas pair inside the container venv. Reinstall the two packages inside /venvs/plaknit (not on the host).

4b. Upgrade plaknit in the persistent venv¶

When a new plaknit release drops, reuse the same venv and cache to minimize downloads:

export VENVBASE=/path/to/project/venvs
export PIPCACHE=/path/to/project/cache
export SIF=otb.sif

singularity exec \
  --bind "$VENVBASE":/venvs \
  --bind "$PIPCACHE":/cache \
  "$SIF" bash -lc '
    source /venvs/plaknit/bin/activate
    pip install --cache-dir /cache --upgrade plaknit
    plaknit --version
  '

5. Activate `plaknit` in terminal:¶

Run this line in your terminal interface to activate your plaknit environment:

1	`source /path/to/venvs/plaknit/lib/activate`

6. Example processing script:¶

# set these to the paths on the host filesystem
export PROJECT_DIR=/path/to/project
export RUN_TAG=aug_2019
export TILES=$PROJECT_DIR/data/strips/$RUN_TAG          # GeoTIFF strips/tiles
export UDMS=$PROJECT_DIR/data/udms/$RUN_TAG            # matching UDM
export OUTDIR=$PROJECT_DIR/output        # mosaic
export VENVBASE=$PROJECT_DIR/venvs       # contains the env
export SCRATCH=${SLURM_TMPDIR:-/tmp}       # fast scratch space
export SIF=otb.sif   # OTB-enabled image

singularity exec \
  --bind "$TILES":/data/strips/$RUN_TAG \
  --bind "$UDMS":/data/udms/$RUN_TAG \
  --bind "$OUTDIR":/data/output \
  --bind "$SCRATCH":/localscratch \
  --bind "$VENVBASE":/venvs \
  "$SIF" bash -lc '
    export OTB_APPLICATION_PATH=/app/otb/lib/otb/applications
    export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=${SLURM_CPUS_PER_TASK:-8}
    export OTB_MAX_RAM_HINT=131072
    export GDAL_CACHEMAX=4096
    mkdir -p /localscratch/tmp

    /venvs/plaknit/bin/plaknit mosaic \
      --inputs $TILES \
      --udms $UDMS \
      --output $OUTDIR/${RUN_TAG}.tif \
      --tmpdir /localscratch/tmp \
      --jobs ${SLURM_CPUS_PER_TASK:-8} \
      --ram 191072
  '

Random Forest classification (train + predict)¶

This Singularity/Apptainer template mirrors the mosaic example but calls plaknit classify. The CLI accepts one or more --image paths; when bands live in separate TIFFs, you can pass them directly (or repeat --image) or build a VRT first (for example gdalbuildvrt stack.vrt band1.tif band2.tif ...). Use --band-indices (1-based) to select a subset of stacked bands.

#!/bin/bash
#SBATCH --job-name=plaknit-classify
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=04:00:00
#SBATCH --output=plaknit_classify_%j.log

# Host paths
export BASE=/path/to/project
export PROJECT=$BASE/rf
export VENVBASE=$BASE/venvs
export PIPCACHE=$BASE/cache/pip
export SCRATCH=${SLURM_TMPDIR:-/tmp}
export SIF=$BASE/otb.sif                             # any GDAL+Python image works; OTB not required for classify

singularity exec \
  --bind "$PROJECT/layers":/layers \
  --bind "$PROJECT/model":/model \
  --bind "$PROJECT/out":/out \
  --bind "$PROJECT":/project \
  --bind "$VENVBASE":/venvs \
  --bind "$PIPCACHE":/pipcache \
  --bind "$SCRATCH":/localscratch \
  "$SIF" bash -lc '

    export PATH=/venvs/plaknit/bin:$PATH

    # Train (optional; skip if model already exists)
    plaknit classify train \
      --image /layers/stack.vrt /layers/mosaic.tif \
      --labels /project/training_points.gpkg \
      --label-column class \
      --model-out /model/rf_model.joblib

    # Predict
    plaknit classify predict \
      --image /layers/stack.vrt /layers/mosaic.tif \
      --model /model/rf_model.joblib \
      --output /out/classification.tif \
      --block-shape 512 512 \
      --jobs 8 \
      --smooth mrf \
      --beta 1.0 \
      --neighborhood 4 \
      --icm-iters 3 \
      --block-overlap 0
  '

7. Validation checklist¶

[ ] pip --version runs successfully inside the container.
[ ] /venvs/plaknit/bin/plaknit --version prints the expected release.
[ ] /venvs/plaknit/bin/python -c "import fiona, geopandas; print(fiona.__version__, geopandas.__version__)" works for classify.
[ ] /data/strips, /data/udms, and /data/output are visible inside the job.
[ ] The output mosaic (for example final_mosaic.tif) lands in $OUTDIR.

8. Summary¶

You now have a reproducible approach that:

Stores all Python artifacts under your project directory.
Uses an OTB-enabled container with a persistent plaknit venv.
Supports both interactive validation and SLURM batch execution.
Avoids $HOME pollution and keeps dependencies isolated.

If your site ships different modules or mount points, adjust the bind paths and SLURM directives but keep the same persistent-venv pattern.