Skip to content

get-preprocessed-dataset-openimages

Automatically generated README for this automation recipe: get-preprocessed-dataset-openimages

Category: AI/ML datasets

License: Apache 2.0

  • Notes from the authors, contributors and users: README-extra

  • CM meta description for this script: _cm.json

  • Output cached? True

Reuse this script in your project

Install MLCommons CM automation meta-framework

Pull CM repository with this automation recipe (CM script)

cm pull repo mlcommons@cm4mlops

cmr "get dataset openimages open-images object-detection preprocessed" --help

Run this script

Run this script via CLI
cm run script --tags=get,dataset,openimages,open-images,object-detection,preprocessed[,variations] [--input_flags]
Run this script via CLI (alternative)
cmr "get dataset openimages open-images object-detection preprocessed [variations]" [--input_flags]
Run this script from Python
import cmind

r = cmind.access({'action':'run'
              'automation':'script',
              'tags':'get,dataset,openimages,open-images,object-detection,preprocessed'
              'out':'con',
              ...
              (other input keys for this script)
              ...
             })

if r['return']>0:
    print (r['error'])
Run this script via Docker (beta)
cm docker script "get dataset openimages open-images object-detection preprocessed[variations]" [--input_flags]

Variations

  • No group (any combination of variations can be selected)

    Click here to expand this section.

    • _filter
    • _for.retinanet.onnx
      • ENV variables:
        • CM_ML_MODEL_NAME: retinanet
        • CM_DATASET_SUBTRACT_MEANS: 1
        • CM_DATASET_GIVEN_CHANNEL_MEANS: 0.485 0.456 0.406
        • CM_DATASET_GIVEN_CHANNEL_STDS: 0.229 0.224 0.225
        • CM_DATASET_NORMALIZE_DATA: 0
        • CM_DATASET_NORMALIZE_LOWER: 0.0
        • CM_DATASET_NORMALIZE_UPPER: 1.0
        • CM_DATASET_CONVERT_TO_BGR: 0
        • CM_DATASET_CROP_FACTOR: 100.0
    • _nvidia
      • ENV variables:
        • CM_PREPROCESSING_BY_NVIDIA: yes
    • _quant-offset.#
    • _quant-scale.#

  • Group "annotations"

    Click here to expand this section.

    • _custom-annotations
    • _default-annotations (default)

  • Group "dataset-count"

    Click here to expand this section.

    • _50 (default)
      • ENV variables:
        • CM_DATASET_SIZE: 50
    • _500
      • ENV variables:
        • CM_DATASET_SIZE: 500
    • _full
    • _size.#
      • ENV variables:
        • CM_DATASET_SIZE: #

  • Group "dataset-layout"

    Click here to expand this section.

    • _NCHW (default)
      • ENV variables:
        • CM_DATASET_DATA_LAYOUT: NCHW
    • _NHWC
      • ENV variables:
        • CM_DATASET_DATA_LAYOUT: NHWC

  • Group "dataset-precision"

    Click here to expand this section.

    • _fp32 (default)
      • ENV variables:
        • CM_DATASET_DTYPE: fp32
        • CM_DATASET_INPUT_DTYPE: fp32
        • CM_DATASET_QUANTIZE: 0
        • CM_DATASET_CONVERT_TO_UNSIGNED: 0
    • _int8
      • ENV variables:
        • CM_DATASET_DTYPE: int8
        • CM_DATASET_INPUT_DTYPE: fp32
        • CM_DATASET_QUANTIZE: 1
        • CM_DATASET_CONVERT_TO_UNSIGNED: 0
    • _uint8
      • ENV variables:
        • CM_DATASET_DTYPE: uint8
        • CM_DATASET_INPUT_DTYPE: fp32
        • CM_DATASET_QUANTIZE: 1
        • CM_DATASET_CONVERT_TO_UNSIGNED: 1

  • Group "dataset-type"

    Click here to expand this section.

    • _calibration
      • ENV variables:
        • CM_DATASET_PATH: <<<CM_CALIBRATION_DATASET_PATH>>>
        • CM_DATASET_ANNOTATIONS_FILE_PATH: <<<CM_DATASET_CALIBRATION_ANNOTATIONS_FILE_PATH>>>
        • CM_DATASET_TYPE: calibration
    • _validation (default)
      • ENV variables:
        • CM_DATASET_TYPE: validation

  • Group "extension"

    Click here to expand this section.

    • _npy
      • ENV variables:
        • CM_DATASET_PREPROCESSED_EXTENSION: npy
    • _raw
      • ENV variables:
        • CM_DATASET_PREPROCESSED_EXTENSION: raw
    • _rgb32
      • ENV variables:
        • CM_DATASET_PREPROCESSED_EXTENSION: rgb32
    • _rgb8
      • ENV variables:
        • CM_DATASET_PREPROCESSED_EXTENSION: rgb8

  • Group "filter-size"

    Click here to expand this section.

    • _filter-size.#

  • Group "interpolation-method"

    Click here to expand this section.

    • _inter.area
      • ENV variables:
        • CM_DATASET_INTERPOLATION_METHOD: INTER_AREA
    • _inter.linear
      • ENV variables:
        • CM_DATASET_INTERPOLATION_METHOD: INTER_LINEAR

  • Group "preprocessing-source"

    Click here to expand this section.

    • _generic-preprocessor
      • ENV variables:
        • CM_DATASET_REFERENCE_PREPROCESSOR: 0
    • _mlcommons-reference-preprocessor (default)
      • ENV variables:
        • CM_DATASET_REFERENCE_PREPROCESSOR: 1
Default variations

_50,_NCHW,_default-annotations,_fp32,_mlcommons-reference-preprocessor,_validation

Script flags mapped to environment

  • --dir=value → CM_DATASET_PREPROCESSED_PATH=value
  • --threads=value → CM_NUM_PREPROCESS_THREADS=value

Default environment

These keys can be updated via --env.KEY=VALUE or env dictionary in @input.json or using script flags.

  • CM_DATASET: OPENIMAGES
  • CM_DATASET_DTYPE: fp32
  • CM_DATASET_INPUT_SQUARE_SIDE: 800
  • CM_DATASET_CROP_FACTOR: 100.0
  • CM_DATASET_QUANT_SCALE: 1
  • CM_DATASET_QUANTIZE: 0
  • CM_DATASET_QUANT_OFFSET: 0

Native script being run

Script output

cmr "get dataset openimages open-images object-detection preprocessed [variations]" [--input_flags] -j