ma-九游平台

执行ma-cli dli-job get-job查询dli spark作业列表或单个作业详情。

ma-cli dli-job get-job -h
usage: ma-cli dli-job get-job [options]
  get dli spark details.
  example:
  # get dli spark job details by job name
  ma-cli dli-job get-job -n ${job_name}
  # get dli spark job details by job id
  ma-cli dli-job get-job -i ${job_id}
  # get dli spark job list
  ma-cli dli-job get-job --page-size 5 --page-num 1
options:
  -i, --job-id text               get dli spark job details by job id.
  -n, --job-name text             get dli spark job details by job name.
  -pn, --page-num integer range   specify which page to query.  [x>=1]
  -ps, --page-size integer range  the maximum number of results for this query.  [x>=1]
  -v, --verbose                   show detailed information about dli spark job details.
  -c, --config-file path          configure file path for authorization.
  -d, --debug                     debug mode. shows full stack trace when error occurs.
  -p, --profile text              cli connection profile to use. the default profile is "default".
  -h, -h, --help                  show this message and exit.

示例：查询dli spark所有作业。

ma-cli dli-job get-job

执行ma-cli dli-job submit命令提交dli spark作业。

ma-cli dli-job submit命令需要指定一个位置参数yaml_file表示作业的配置文件路径，如果不指定该参数，则表示配置文件为空。配置文件是一个yaml格式的文件，里面的参数就是命令的option参数。此外，如果用户在命令行中同时指定yaml_file配置文件和option参数，命令行中指定的option参数的值将会覆盖配置文件相同的值。

命令参数预览

ma-cli dli-job submit -h
usage: ma-cli dli-job submit [options] [yaml_file]...
  submit dli spark job.
  example:
  ma-cli dli-job submit  --name test-spark-from-sdk
                          --file test/sub_dli_task.py
                          --obs-bucket dli-bucket
                          --queue dli_test
                          --spark-version 2.4.5
                          --driver-cores 1
                          --driver-memory 1g
                          --executor-cores 1
                          --executor-memory 1g
                          --num-executors 1
options:
  --file text                    python file or app jar.
  -cn, --class-name text         your application's main class (for java / scala apps).
  --name text                    job name.
  --image text                   full swr custom image path.
  --queue text                   execute queue name.
  -obs, --obs-bucket text        dli obs bucket to save logs.
  -sv, --spark-version text      spark version.
  -st, --sc-type [a|b|c]         compute resource type.
  --feature [basic|custom|ai]    type of the spark image used by a job (default: basic).
  -ec, --executor-cores integer  executor cores.
  -em, --executor-memory text    executor memory (eg. 2g/2048mb).
  -ne, --num-executors integer   executor number.
  -dc, --driver-cores integer    driver cores.
  -dm, --driver-memory text      driver memory (eg. 2g/2048mb).
  --conf text                    arbitrary spark configuration property (eg. ).
  --resources text               resources package path.
  --files text                   files to be placed in the working directory of each executor.
  --jars text                    jars to include on the driver and executor class paths.
  -pf, --py-files text           python files to place on the pythonpath for python apps.
  --groups text                  user group resources.
  --args text                    spark batch job parameter args.
  -q, --quiet                    exit without waiting after submit successfully.
  -c, --config-file path         configure file path for authorization.
  -d, --debug                    debug mode. shows full stack trace when error occurs.
  -p, --profile text             cli connection profile to use. the default profile is "default".
  -h, -h, --help                 show this message and exit.

yaml文件预览

# dli-demo.yaml
name: test-spark-from-sdk
file: test/sub_dli_task.py
obs-bucket: ${your_bucket}
queue: dli_notebook 
spark-version: 2.4.5
driver-cores: 1
driver-memory: 1g
executor-cores: 1
executor-memory: 1g
num-executors: 1
## [optional] 
jars:
  - ./test.jar
  - obs://your-bucket/jars/test.jar
  - your_group/test.jar
## [optional] 
files:
  - ./test.csv
  - obs://your-bucket/files/test.csv
  - your_group/test.csv
## [optional] 
python-files:
  - ./test.py
  - obs://your-bucket/files/test.py
  - your_group/test.py
## [optional] 
resources:
  - name: your_group/test.py
    type: pyfile
  - name: your_group/test.csv
    type: file
  - name: your_group/test.jar
    type: jar
  - name: ./test.py
    type: pyfile
  - name: obs://your-bucket/files/test.py
    type: pyfile
## [optional]
groups:
  - group1
  - group2

指定options参数提交dli spark作业示例：

$ ma-cli dli-job submit --name test-spark-from-sdk \
                        --file test/sub_dli_task.py \
                        --obs-bucket ${your_bucket} \
                        --queue dli_test \
                        --spark-version 2.4.5 \
                        --driver-cores 1 \
                        --driver-memory 1g \
                        --executor-cores 1 \
                        --executor-memory 1g \
                        --num-executors 1 

示例

• 通过yaml_file文件提交dli spark作业。

  $ma-cli dli-job submit dli_job.yaml

  

• 指定命令行options参数提交dli spark作业。

  $ma-cli dli-job submit --name test-spark-from-sdk \
  >                         --file test/jumpstart-trainingjob-gallery-pytorch-sample.ipynb \
  >                         --queue dli_ma_notebook \
  >                         --spark-version 2.4.5 \
  >                         --driver-cores 1 \
  >                         --driver-memory 1g \
  >                         --executor-cores 1 \
  >                         --executor-memory 1g \
  >                         --num-executors 1 

  

执行ma-cli dli-job get-log命令查询dli spark作业后台的日志。

$ ma-cli dli-job get-log -h
usage: ma-cli dli-job get-log [options]
  get dli spark job log details.
  example:
  # get job log by job id
  ma-cli dli-job get-log --job-id ${job_id}
options:
  -i, --job-id text       get dli spark job details by job id.  [required]
  -c, --config-file text  configure file path for authorization.
  -d, --debug             debug mode. shows full stack trace when error occurs.
  -p, --profile text      cli connection profile to use. the default profile is "default".
  -h, -h, --help          show this message and exit.

示例：查询指定作业id的dli spark作业运行日志。

ma-cli dli-job get-log --job-id ${your_job_id}

执行ma-cli dli-job get-queue命令查询dli队列。

ma-cli dli-job get-queue -h
usage: ma-cli dli-job get-queue [options]
  get dli queues info.
  example:
  # get dli queue details by queue name
  ma-cli dli-job get-queue --queue-name $queue_name}
options:
  -pn, --page-num integer range   specify which page to query.  [x>=1]
  -ps, --page-size integer range  the maximum number of results for this query.  [x>=1]
  -n, --queue-name text           get dli queue details by queue name.
  -t, --queue-type [sql|general|all]
                                  dli queue type (default "all").
  -tags, --tags text              get dli queues by tags.
  -c, --config-file path          configure file path for authorization.
  -d, --debug                     debug mode. shows full stack trace when error occurs.
  -p, --profile text              cli connection profile to use. the default profile is "default".
  -h, -h, --help                  show this message and exit.

示例：查询队列名为“dli_ma_notebook”的队列信息。

ma-cli dli-job get-queue --queue-name dli_ma_notebook

执行ma-cli dli-job get-resource命令获取dli资源详细信息，如资源名称，资源类型等。

$ ma-cli dli-job get-resource -h
usage: ma-cli dli-job get-resource [options]
  get dli resource info.
  example:
  # get dli resource details by resource name
  ma-cli dli-job get-resource --resource-name ${resource_name}
options:
  -n, --resource-name text        get dli resource details by resource name.
  -k, --kind [jar|pyfile|file|modelfile]
                                  dli resources type.
  -g, --group text                get dli resources by group.
  -tags, --tags text              get dli resources by tags.
  -c, --config-file text          configure file path for authorization.
  -d, --debug                     debug mode. shows full stack trace when error occurs.
  -p, --profile text              cli connection profile to use. the default profile is "default".
  -h, -h, --help                  show this message and exit.

示例：查询所有dli分组资源信息。

ma-cli dli-job get-resource

ma-cli dli-job upload命令支持将本地文件或obs文件上传到dli资源组。

$ ma-cli dli-job upload -h
usage: ma-cli dli-job upload [options] paths...
  upload dli resource.
  tips: --obs-path is need when upload local file.
  example:
  # upload an obs path to dli resource
  ma-cli dli-job upload obs://your-bucket/test.py -g test-group --kind pyfile
  # upload a local path to dli resource
  ma-cli dli-job upload ./test.py -g test-group -obs ${your-bucket} --kind pyfile
  # upload local path and obs path to dli resource
  ma-cli dli-job upload ./test.py obs://your-bucket/test.py -g test-group -obs ${your-bucket}
options:
  -k, --kind [jar|pyfile|file]  dli resources type.
  -g, --group text              dli resources group.
  -tags, --tags text            dli resources tags, follow --tags `key1`=`value1`.
  -obs, --obs-bucket text       obs bucket for upload local file.
  -async, --is-async            whether to upload resource packages in asynchronous mode. the default value is false.
  -c, --config-file text        configure file path for authorization.
  -d, --debug                   debug mode. shows full stack trace when error occurs.
  -p, --profile text            cli connection profile to use. the default profile is "default".
  -h, -h, --help                show this message and exit.

示例

• 上传本地文件到dli分组资源

  ma-cli dli-job upload ./test.py -obs ${your-bucket} --kind pyfile

  

• 上传obs文件到dli分组资源

  ma-cli dli-job upload obs://your-bucket/test.py --kind pyfile

  

执行ma-cli dli-job stop命令停止dli spark作业。

$ ma-cli dli-job stop -h
usage: ma-cli dli-job stop [options]
  stop dli spark job by job id.
  example:
  stop training job by job id
  ma-cli dli-job stop --job-id ${job_id}
options:
  -i, --job-id text       get dli spark job event by job id.  [required]
  -y, --yes               confirm stop operation.
  -c, --config-file text  configure file path for authorization.
  -d, --debug             debug mode. shows full stack trace when error occurs.
  -p, --profile text      cli connection profile to use. the default profile is "default".
  -h, -h, --help          show this message and exit.

示例

ma-cli dli-job stop -i ${your_job_id}