Skip to content

app-loadgen-generic-python

Automatically generated README for this automation recipe: app-loadgen-generic-python

Category: Modular MLPerf inference benchmark pipeline

License: Apache 2.0

Developers: Gaz Iqbal, Arjun Suresh, Grigori Fursin * Notes from the authors, contributors and users: README-extra

  • CM meta description for this script: _cm.yaml
  • Output cached? False

Reuse this script in your project

Install MLCommons CM automation meta-framework

Pull CM repository with this automation recipe (CM script)

cm pull repo mlcommons@cm4mlops

cmr "python app generic loadgen" --help

Run this script

Run this script via CLI
cm run script --tags=python,app,generic,loadgen[,variations] [--input_flags]
Run this script via CLI (alternative)
cmr "python app generic loadgen [variations]" [--input_flags]
Run this script from Python
import cmind

r = cmind.access({'action':'run'
              'automation':'script',
              'tags':'python,app,generic,loadgen'
              'out':'con',
              ...
              (other input keys for this script)
              ...
             })

if r['return']>0:
    print (r['error'])
Run this script via Docker (beta)
cm docker script "python app generic loadgen[variations]" [--input_flags]

Variations

  • No group (any combination of variations can be selected)

    Click here to expand this section.

    • _cmc
      • ENV variables:
        • CM_CUSTOM_MODEL_CMC: True
    • _huggingface
      • ENV variables:
        • CM_CUSTOM_MODEL_SOURCE: huggingface
    • _model-stub.#
      • ENV variables:
        • CM_ML_MODEL_STUB: #

  • Group "backend"

    Click here to expand this section.

    • _onnxruntime (default)
      • ENV variables:
        • CM_MLPERF_BACKEND: onnxruntime
    • _pytorch
      • ENV variables:
        • CM_MLPERF_BACKEND: pytorch

  • Group "device"

    Click here to expand this section.

    • _cpu (default)
      • ENV variables:
        • CM_MLPERF_DEVICE: cpu
        • CM_MLPERF_EXECUTION_PROVIDER: CPUExecutionProvider
    • _cuda
      • ENV variables:
        • CM_MLPERF_DEVICE: gpu
        • CM_MLPERF_EXECUTION_PROVIDER: CUDAExecutionProvider

  • Group "models"

    Click here to expand this section.

    • _custom
      • ENV variables:
        • CM_MODEL: custom
    • _resnet50
      • ENV variables:
        • CM_MODEL: resnet50
    • _retinanet
      • ENV variables:
        • CM_MODEL: retinanet
Default variations

_cpu,_onnxruntime

Input Flags

  • --modelpath: Full path to file with model weights
  • --modelcodepath: (for PyTorch models) Full path to file with model code and cmc.py
  • --modelcfgpath: (for PyTorch models) Full path to JSON file with model cfg
  • --modelsamplepath: (for PyTorch models) Full path to file with model sample in pickle format
  • --ep: ONNX Execution provider
  • --scenario: MLPerf LoadGen scenario
  • --samples: Number of samples (2)
  • --runner: MLPerf runner
  • --execmode: MLPerf exec mode
  • --output_dir: MLPerf output directory
  • --concurrency: MLPerf concurrency
  • --intraop: MLPerf intra op threads
  • --interop: MLPerf inter op threads

Script flags mapped to environment

  • --concurrency=value → CM_MLPERF_CONCURRENCY=value
  • --ep=value → CM_MLPERF_EXECUTION_PROVIDER=value
  • --execmode=value → CM_MLPERF_EXEC_MODE=value
  • --interop=value → CM_MLPERF_INTEROP=value
  • --intraop=value → CM_MLPERF_INTRAOP=value
  • --loadgen_duration_sec=value → CM_MLPERF_LOADGEN_DURATION_SEC=value
  • --loadgen_expected_qps=value → CM_MLPERF_LOADGEN_EXPECTED_QPS=value
  • --modelcfg=value → CM_ML_MODEL_CFG=value
  • --modelcfgpath=value → CM_ML_MODEL_CFG_WITH_PATH=value
  • --modelcodepath=value → CM_ML_MODEL_CODE_WITH_PATH=value
  • --modelpath=value → CM_ML_MODEL_FILE_WITH_PATH=value
  • --modelsamplepath=value → CM_ML_MODEL_SAMPLE_WITH_PATH=value
  • --output_dir=value → CM_MLPERF_OUTPUT_DIR=value
  • --runner=value → CM_MLPERF_RUNNER=value
  • --samples=value → CM_MLPERF_LOADGEN_SAMPLES=value
  • --scenario=value → CM_MLPERF_LOADGEN_SCENARIO=value

Default environment

These keys can be updated via --env.KEY=VALUE or env dictionary in @input.json or using script flags.

  • CM_MLPERF_EXECUTION_MODE: parallel
  • CM_MLPERF_BACKEND: onnxruntime

Native script being run

Script output

cmr "python app generic loadgen [variations]" [--input_flags] -j