ma-九游平台

ma-cli提供了一些常用的镜像构建模板，模板中包含了在modelarts notebook上进行dockerfile开发的牵引指导。

$ ma-cli image get-template -h
usage: ma-cli image get-template [options]
  list build-in dockerfile templates.
  example:
  # list build-in dockerfile templates
  ma-cli image get-template [--filer ]  [--page-num ] [--page-size ]
options:
  --filter text                   filter by keyword.
  -pn, --page-num integer range   specify which page to query.  [x>=1]
  -ps, --page-size integer range  the maximum number of results for this query.  [x>=1]
  -d, --debug                     debug mode. shows full stack trace when error occurs.
  -p, --profile text              cli connection profile to use. the default profile is "default".
  -h, -h, --help                  show this message and exit.
(pytorch-1.4) [ma-user work]$

示例：查看镜像构建模板。

ma-cli image get-template

ma-cli可以使用add-template命令将镜像模板加载到指定文件夹下，默认路径为当前命令所在的路径。

比如${current_dir}/.ma/${template_name}/。也可以通过--dest命令指定保存的路径。当保存的路径已经有同名的模板文件夹时，可以使用--force | -f参数进行强制覆盖。

$ ma-cli image add-template -h
usage: ma-cli image add-template [options] template_name
  add buildin dockerfile templates into disk.
  example:
  # list build-in dockerfile templates
  ma-cli image add-template customize_from_ubuntu_18.04_to_modelarts --force
options:
  --dst text          target save path.
  -f, --force         override templates that has been installed.
  -d, --debug         debug mode. shows full stack trace when error occurs.
  -p, --profile text  cli connection profile to use. the default profile is "default".
  -h, -h, --help      show this message and exit.

示例：加载customize_from_ubuntu_18.04_to_modelarts镜像构建模板。

ma-cli image add-template customize_from_ubuntu_18.04_to_modelarts

dockerfile一般需要提供一个基础镜像的地址，目前支持从docker hub等开源镜像仓拉取公开镜像，以及swr的公开或私有镜像。其中ma-cli提供了查询modelarts预置镜像和用户已注册镜像列表及swr地址。

$ma-cli image get-image -h
usage: ma-cli image get-image [options]
  get registered image list.
  example:
  # query images by image type and only image id, show name and swr_path
  ma-cli image get-image --type=dedicated
  # query images by image id
  ma-cli image get-image --image-id ${image_id}
  # query images by image type and show more information
  ma-cli image get-image --type=dedicated -v
  # query images by image name
  ma-cli image get-image --filter=torch
options:
  -t, --type [build_in|dedicated|all]
                                  image type(default all)
  -f, --filter text               image name to filter
  -v, --verbose                   show detailed information on image.
  -i, --image-id text             get image details by image id
  -n, --image-name text           get image details by image name
  -wi, --workspace-id text        the workspace where you want to query image(default "0")
  -pn, --page-num integer range   specify which page to query  [x>=1]
  -ps, --page-size integer range  the maximum number of results for this query  [x>=1]
  -c, --config-file path          configure file path for authorization.
  -d, --debug                     debug mode. shows full stack trace when error occurs.
  -p, --profile text              cli connection profile to use. the default profile is "default".
  -h, -h, --help                  show this message and exit.

示例：查询modelarts已注册的自定义镜像。

ma-cli image get-image --type=dedicated

使用ma-cli image build命令基于指定的dockerfile进行镜像构建，仅支持在modelarts notebook里使用该命令。

$ ma-cli image build -h
usage: ma-cli image build [options] file_path
  build docker image in notebook.
  example:
  # build a image and push to swr
  ma-cli image build .ma/customize_from_ubuntu_18.04_to_modelarts/dockerfile -swr my_organization/my_image:0.0.1
  # build a image and push to swr, dockerfile context path is current dir
  ma-cli image build .ma/customize_from_ubuntu_18.04_to_modelarts/dockerfile -swr my_organization/my_image:0.0.1 -context .
  # build a local image and save to local path and obs
  ma-cli image build .ma/customize_from_ubuntu_18.04_to_modelarts/dockerfile --target ./build.tar --obs_path obs://bucket/object --swr-path my_organization/my_image:0.0.1
options:
  -t, --target text       name and optionally a tag in the 'name:tag' format.
  -swr, --swr-path text   swr path without swr endpoint, eg:organization/image:tag.  [required]
  --context directory     build context path.
  -arg, --build-arg text  build arg for dockerfile.
  -obs, --obs-path text   obs path to save local built image.
  -f, --force             force to overwrite the existing swr image with the same name and tag.
  -c, --config-file path  configure file path for authorization.
  -d, --debug             debug mode. shows full stack trace when error occurs.
  -p, --profile text      cli connection profile to use. the default profile is "default".
  -h, -h, --help          show this message and exit.

示例：在modelarts notebook里进行镜像构建。

ma-cli image build .ma/customize_from_ubuntu_18.04_to_modelarts/dockerfile -swr notebook_test/my_image:0.0.1

其中“.ma/customize_from_ubuntu_18.04_to_modelarts/dockerfile”为dockerfile文件所在路径，“notebook_test/my_image:0.0.1”为构建的新镜像的swr路径。

使用ma-cli image df命令查询镜像构建缓存，仅支持在modelarts notebook里使用该命令。

$ ma-cli image df -h
usage: ma-cli image df [options]
  query disk usage used by image-building in notebook.
  example:
  # query image disk usage
  ma-cli image df
options:
  -v, --verbose       show detailed information on disk usage.
  -d, --debug         debug mode. shows full stack trace when error occurs.
  
  -h, -h, --help      show this message and exit.

ma-cli image df

ma-cli image df --verbose

使用ma-cli image prune命令清理镜像构建缓存，仅支持在modelarts notebook里使用该命令。

$ ma-cli image prune -h
usage: ma-cli image prune [options]
  prune image build cache by image-building in notebook.
  example:
  # prune image build cache
  ma-cli image prune
options:
  -ks, --keep-storage integer  amount of disk space to keep for cache below this limit (in mb) (default: 0).
  -kd, --keep-duration text    keep cache newer than this limit, support second(s), minute(m) and hour(h) (default:   0s).
  -v, --verbose                show more verbose output.
  -d, --debug                  debug mode. shows full stack trace when error occurs.  
  -h, -h, --help               show this message and exit.

示例：清理保留1mb镜像缓存。

ma-cli image prune -ks 1

调试完成后，使用ma-cli image register命令将新镜像注册到modelarts镜像管理服务中，进而在能够在modelarts中使用该镜像。

$ma-cli image register -h
usage: ma-cli image register [options]
  register image to modelarts.
  example:
  # register image into modelarts service
  ma-cli image register --swr-path=xx
  # share swr image to dli service
  ma-cli image register -swr xx -td
  # register image into modelarts service and specify architecture to be 'aarch64'
  ma-cli image register --swr-path=xx --arch aarch64
options:
  -swr, --swr-path text           swr path without swr endpoint, eg:organization/image:tag.  [required]
  -a, --arch [x86_64|aarch64]     image architecture (default: x86_64).
  -s, --service [notebook|modelbox]
                                  services supported by this image(default notebook).
  -rs, --resource-category [cpu|gpu|ascend]
                                  the resource category supported by this image (default: cpu and gpu).
  -wi, --workspace-id text        the workspace to register this image (default: "0").
  -v, --visibility [public|private]
                                  public: every user can use this image. private: only image owner can use this image (default: private).
  -td, --to-dli                   register swr image to dli, which will share swr image to dli service.
  -d, --description text          image description (default: "").
  -c, --config-file path          configure file path for authorization.
  -d, --debug                     debug mode. shows full stack trace when error occurs.
  -p, --profile text              cli connection profile to use. the default profile is "default".
  -h, -h, --help                  show this message and exit.

示例：注册swr镜像到modelarts。

ma-cli image register --swr-path=xx

使用ma-cli image unregister命令将注册的镜像从modelarts中删除。

$ ma-cli image unregister -h
usage: ma-cli image unregister [options]
  unregister image from modelarts.
  example:
  # unregister image
  ma-cli image unregister --image-id=xx
  # unregister image and delete it from swr
  ma-cli image unregister --image-id=xx -d
options:
  -i, --image-id text     unregister image details by image id.  [required]
  -d, --delete-swr-image  delete the image from swr.
  -c, --config-file path  configure file path for authorization.
  -d, --debug             debug mode. shows full stack trace when error occurs.
  -p, --profile text      cli connection profile to use. the default profile is "default".
  -h, -h, --help          show this message and exit.

ma-cli支持在ecs上调试swr镜像是否可以在modelarts开发空间中运行，发现镜像中可能存在的问题。