ma-九游平台

使用ma-cli ma-job get-job命令查询modelarts训练作业

使用ma-cli ma-job get-job命令可以查看训练作业列表或某个作业详情。

$ ma-cli ma-job get-job -h
usage: ma-cli ma-job get-job [options]
  get job details.
  example:
  # get train job details by job name
  ma-cli ma-job get-job -n ${job_name}
  # get train job details by job id
  ma-cli ma-job get-job -i ${job_id}
  # get train job list
  ma-cli ma-job get-job --page-size 5 --page-num 1
options:
  
  -i, --job-id text               get training job details by job id.
  -n, --job-name text             get training job details by job name.
  -pn, --page-num integer         specify which page to query.  [x>=1]
  -ps, --page-size integer range  the maximum number of results for this query.  [1<=x<=50]
  -v, --verbose                   show detailed information about training job details.
  -c, --config-file text          configure file path for authorization.
  -d, --debug                     debug mode. shows full stack trace when error occurs.
  -p, --profile text              cli connection profile to use. the default profile is "default".
  -h, -h, --help                  show this message and exit.

• 示例：查询指定任务id的训练作业。

  ma-cli ma-job get-job -i b63e90xxx

  

• 示例：根据任务名称关键字“auto”过滤训练作业。

  ma-cli ma-job get-job -n auto

  

使用ma-cli ma-job submit命令提交modelarts训练作业

执行ma-cli ma-job submit命令提交modelarts训练作业。

ma-cli ma-job submit命令需要指定一个位置参数yaml_file表示作业的配置文件路径，如果不指定该参数，则表示配置文件为空。配置文件是一个yaml格式的文件，里面的参数就是命令的option参数。此外，如果用户在命令行中同时指定yaml_file配置文件和option参数，命令行中指定的option参数的值将会覆盖配置文件相同的值。

$ma-cli ma-job submit -h
usage: ma-cli ma-job submit [options] [yaml_file]...
  submit training job.
  example:
  ma-cli ma-job submit --code-dir obs://your_bucket/code/
                       --boot-file main.py
                       --framework-type pytorch
                       --working-dir /home/ma-user/modelarts/user-job-dir/code
                       --framework-version pytorch_1.8.0-cuda_10.2-py_3.7-ubuntu_18.04-x86_64
                       --data-url obs://your_bucket/dataset/
                       --log-url obs://your_bucket/logs/
                       --train-instance-type modelarts.vm.cpu.8u
                       --train-instance-count 1
options:
  --name text                     job name.
  --description text              job description.
  --image-url text                full swr custom image path.
  --uid text                      uid for custom image (default: 1000).
  --working-dir text              modelarts training job working directory.
  --local-code-dir text           modelarts training job local code directory.
  --user-command text             execution command for custom image.
  --pool-id text                  dedicated pool id.
  --train-instance-type text      train worker specification.
  --train-instance-count integer  number of workers.
  --data-url text                 obs path for training data.
  --log-url text                  obs path for training log.
  --code-dir text                 obs path for source code.
  --output text                   training output parameter with obs path.
  --input text                    training input parameter with obs path.
  --env-variables text            env variables for training job.
  --parameters text               training job parameters (only keyword parameters are supported).
  --boot-file text                training job boot file path behinds `code_dir`.
  --framework-type text           training job framework type.
  --framework-version text        training job framework version.
  --workspace-id text             the workspace where you submit training job(default "0")
  --policy [regular|economic|turbo|auto]
                                  training job policy, default is regular.
  --volumes text                  information about the volumes attached to the training job.
  -q, --quiet                     exit without waiting after submit successfully.
  -c, --config-file path          configure file path for authorization.
  -d, --debug                     debug mode. shows full stack trace when error occurs.
  -p, --profile text              cli connection profile to use. the default profile is "default".
  -h, -h, --help                  show this message and exit.

示例：基于modelarts预置镜像提交训练作业

指定命令行options参数提交训练作业

ma-cli ma-job submit --code-dir obs://your-bucket/mnist/code/ \
                  --boot-file main.py \
                  --framework-type pytorch \
                  --working-dir /home/ma-user/modelarts/user-job-dir/code \
                  --framework-version pytorch_1.8.0-cuda_10.2-py_3.7-ubuntu_18.04-x86_64 \
                  --data-url obs://your-bucket/mnist/dataset/mnist/ \
                  --log-url obs://your-bucket/mnist/logs/ \
                  --train-instance-type modelarts.vm.cpu.8u \
                  --train-instance-count 1  \
                  -q

使用预置镜像的train.yaml样例：

# .ma/train.yaml样例（预置镜像）
# pool_id: pool_xxxx
train-instance-type: modelarts.vm.cpu.8u
train-instance-count: 1
data-url: obs://your-bucket/mnist/dataset/mnist/
code-dir: obs://your-bucket/mnist/code/
working-dir: /home/ma-user/modelarts/user-job-dir/code
framework-type: pytorch
framework-version: pytorch_1.8.0-cuda_10.2-py_3.7-ubuntu_18.04-x86_64
boot-file: main.py
log-url: obs://your-bucket/mnist/logs/
##[optional] uncomment to set uid when use custom image mode
uid: 1000
##[optional] uncomment to upload output file/dir to obs from training platform
output:
    - name: output_dir
      obs_path: obs://your-bucket/mnist/output1/
##[optional] uncomment to download input file/dir from obs to training platform
input:
    - name: data_url
      obs_path: obs://your-bucket/mnist/dataset/mnist/
##[optional] uncomment pass hyperparameters
parameters:
    - epoch: 10
    - learning_rate: 0.01
    - pretrained:
##[optional] uncomment to use dedicated pool
pool_id: pool_xxxx
##[optional] uncomment to use volumes attached to the training job
volumes:
  - efs:
      local_path: /xx/yy/zz
      read_only: false
      nfs_server_path: xxx.xxx.xxx.xxx:/

示例：基于自定义镜像创建训练作业

指定命令行options参数提交训练作业

ma-cli ma-job submit --image-url atelier/pytorch_1_8:pytorch_1.8.0-cuda_10.2-py_3.7-ubuntu_18.04-x86_64-20220926104358-041ba2e \
                  --code-dir obs://your-bucket/mnist/code/ \
                  --user-command "export ld_library_path=/usr/local/cuda/compat:$ld_library_path && cd /home/ma-user/modelarts/user-job-dir/code && /home/ma-user/anaconda3/envs/pytorch-1.8/bin/python main.py" \
                  --data-url obs://your-bucket/mnist/dataset/mnist/ \
                  --log-url obs://your-bucket/mnist/logs/ \
                  --train-instance-type modelarts.vm.cpu.8u \
                  --train-instance-count 1  \
                  -q

使用自定义镜像的train.yaml样例：

# .ma/train.yaml样例（自定义镜像）
image-url: atelier/pytorch_1_8:pytorch_1.8.0-cuda_10.2-py_3.7-ubuntu_18.04-x86_64-20220926104358-041ba2e
user-command: export ld_library_path=/usr/local/cuda/compat:$ld_library_path && cd /home/ma-user/modelarts/user-job-dir/code && /home/ma-user/anaconda3/envs/pytorch-1.8/bin/python main.py
train-instance-type: modelarts.vm.cpu.8u
train-instance-count: 1
data-url: obs://your-bucket/mnist/dataset/mnist/
code-dir: obs://your-bucket/mnist/code/
log-url: obs://your-bucket/mnist/logs/
##[optional] uncomment to set uid when use custom image mode
uid: 1000
##[optional] uncomment to upload output file/dir to obs from training platform
output:
    - name: output_dir
      obs_path: obs://your-bucket/mnist/output1/
##[optional] uncomment to download input file/dir from obs to training platform
input:
    - name: data_url
      obs_path: obs://your-bucket/mnist/dataset/mnist/
##[optional] uncomment pass hyperparameters
parameters:
    - epoch: 10
    - learning_rate: 0.01
    - pretrained:
##[optional] uncomment to use dedicated pool
pool_id: pool_xxxx
##[optional] uncomment to use volumes attached to the training job
volumes:
  - efs:
      local_path: /xx/yy/zz
      read_only: false
      nfs_server_path: xxx.xxx.xxx.xxx:/

使用ma-cli ma-job get-log命令查询modelarts训练作业日志

执行ma-cli ma-job get-log命令查询modelarts训练作业日志。

$ ma-cli ma-job get-log -h
usage: ma-cli ma-job get-log [options]
  get job log details.
  example:
  # get job log by job id
  ma-cli ma-job get-log --job-id ${job_id}
options:
  -i, --job-id text       get training job details by job id.  [required]
  -t, --task-id text      get training job details by task id (default "worker-0").
  -c, --config-file text  configure file path for authorization.
  -d, --debug             debug mode. shows full stack trace when error occurs.
  -p, --profile text      cli connection profile to use. the default profile is "default".
  -h, -h, --help          show this message and exit.

示例：查询指定训练作业id的作业日志。

ma-cli ma-job get-log --job-id b63e90baxxx

使用ma-cli ma-job get-event命令查询modelarts训练作业事件

执行ma-cli ma-job get-event命令查看modelarts训练作业事件。

$ ma-cli ma-job get-event -h
usage: ma-cli ma-job get-event [options]
  get job running event.
  example:
  # get training job running event
  ma-cli ma-job get-event --job-id ${job_id}
options:
  -i, --job-id text       get training job event by job id.  [required]
  -c, --config-file text  configure file path for authorization.
  -d, --debug             debug mode. shows full stack trace when error occurs.
  -p, --profile text      cli connection profile to use. the default profile is "default".
  -h, -h, --help          show this message and exit.

示例：查看指定id的训练作业的事件详情等。

ma-cli ma-job get-event --job-id b63e90baxxx

使用ma-cli ma-job get-engine命令查询modelarts训练ai引擎

执行ma-cli ma-job get-engine命令查询modelarts训练使用的ai引擎。

$ ma-cli ma-job get-engine -h
usage: ma-cli ma-job get-engine [options]
  get job engine info.
  example:
  # get training job engines
  ma-cli ma-job get-engine
options:
  -v, --verbose           show detailed information about training engines.
  -c, --config-file text  configure file path for authorization.
  -d, --debug             debug mode. shows full stack trace when error occurs.
  -p, --profile text      cli connection profile to use. the default profile is "default".
  -h, -h, --help          show this message and exit.

示例：查看训练作业的ai引擎。

ma-cli ma-job get-engine

使用ma-cli ma-job get-flavor命令查询modelarts训练资源规格

执行ma-cli ma-job get-flavor命令查询modelarts训练的资源规格。

$ ma-cli ma-job get-flavor -h
usage: ma-cli ma-job get-flavor [options]
  get job flavor info.
  example:
  # get training job flavors
  ma-cli ma-job get-flavor
options:
  -t, --flavor-type [cpu|gpu|ascend]
                                  type of training job flavor.
  -v, --verbose                   show detailed information about training flavors.
  -c, --config-file text          configure file path for authorization.
  -d, --debug                     debug mode. shows full stack trace when error occurs.
  -p, --profile text              cli connection profile to use. the default profile is "default".
  -h, -h, --help                  show this message and exit.

示例：查看训练作业的资源规格及类型。

ma-cli ma-job get-flavor

使用ma-cli ma-job stop命令停止modelarts训练作业

执行ma-cli ma-job stop命令，可停止指定作业id的训练作业。

$ ma-cli ma-job stop -h
usage: ma-cli ma-job stop [options]
  stop training job by job id.
  example:
  stop training job by job id
  ma-cli ma-job stop --job-id ${job_id}
options:
  -i, --job-id text       get training job event by job id.  [required]
  -y, --yes               confirm stop operation.
  -c, --config-file text  configure file path for authorization.
  -d, --debug             debug mode. shows full stack trace when error occurs.
  -p, --profile text      cli connection profile to use. the default profile is "default".
  -h, -h, --help          show this message and exit.

示例：停止运行中的训练作业。

ma-cli ma-job stop --job-id efd3e2f8xxx