Deadline plugin¶

Overview¶

Conductor and Deadline are integrated via a tool-set that sends jobs to Conductor and ensures they only run on Conductor instances. Upload and download tasks are handled by Conductor's upload and download daemons.

One of Conductor's underlying principles is to only spin-up an instance when a specific job or task needs it.

Just as with any other instance in a Deadline managed farm, Conductor instances run the Deadline Worker.

New

The Deadline integration now supports a new mode (native) for submitting jobs. When jobs are submitted in native mode, Deadline hands-off the job to Conductor. No Deadline Worker is run and the jobs can no longer be monitored in Deadline Monitor. The advantage is a much simplified process that doesn't require the Deadline RCS or direct communication between a Worker on Conductor and your Deadline Repo.

Installation¶

If you haven't already done so, Download the Deadline Plugin.

Steps 2-4 require SuperUser access in Deadline Monitor

If submitting jobs to Conductor exclusively in native mode, steps #4, #6 and #7 can be skipped.

Copy plugins to the correct folders:

deadline_plugin/custom/scripts/Jobs/* -> /custom/scripts/Jobs/
deadline_plugin/custom/events/ConductorWorker -> /custom/events/

Install the pre-requisite Conductor python libraries: pip install -r deadline-plugin/requirements.txt
Add the path of the Conductor python libraries (installed in step #2) and deadline_plugin/src to the PYTHONPATH of all your clients or to the python paths in your Deadline repo.

Add the Conductor paths to the MayaCmd plugin. You only need to add for the versions that you intend to use. If you plan to submit with other packages, please reach out to Conductor's support team with the names and version and we'll reply with the correct paths.

        /opt/autodesk/maya-io/2019/maya-io2019.SP0/bin/Render
        /opt/autodesk/maya-io/2019/maya-io2019.SP1/bin/Render
        /opt/autodesk/maya-io/2019/maya-io2019.SP2/bin/Render
        /opt/autodesk/maya-io/2020/maya-io2020.SP0/bin/Render
        /opt/autodesk/maya-io/2022/maya-io2022.SP0/bin/Render
        /opt/autodesk/maya-io/2022/maya-io2022.SP3/bin/Render
        /opt/autodesk/maya-io/2023/maya-io2023.SP1/bin/Render
        /opt/autodesk/maya-io/2023/maya-io2023.SP3/bin/Render
        /opt/autodesk/maya-io/2024/maya-io2024.SP1/bin/Render

Add submit_to_conductor.py as a Job menu script. The path to the icon is deadline_plugin/custom/scripts/General/conductor_logo.ico
The Deadline Workers running on Conductor instances need to connect to your repository via a Remote Connection Server (RCS). See the Deadline documentation for more details. This will often require you to open ports in your firewall to ports 8000 or 4433 (TLS) - however, the exact ports are configurable via the Deadline config.

To connect to your Deadline RCS, you will need the RCS's hostname/IP, port, and the client certificate (.pfx). To provide theses values, you can either set the environment variables CONDUCTOR_DEADLINE_PROXY and CONDUCTOR_DEADLINE_SSL_CERTIFICATE before launching the Monitor or modify the code directly in send_to_conductor.py. Please note that client certificates with a passphrase cannot be configured through the web interface. Please contact Conductor's support team for more information.

Usage¶

Prerequisites¶

The Conductor upload daemon must be running
The Conductor download daemon must be running
A dependency sidecar exists for the given job
You have contacted Conductor's support team, to request Deadline support on your account.

Sending Jobs to Conductor¶

From the Deadline Monitor, select one or multiple jobs.
From the context menu, select Scripts->Send to Conductor
A dialog will appear where you can modify the default options for a job
To submit a native job and have Conductor run the job directly without a Deadline Worker, selected the Native check-box. Once submitted, the job will have to be monitored and managed in the Conductor dashboard. It's left to the user to gracefully disable or remove the job from Deadline.
Once you click Submit, the job is shipped to Conductor
The dialog reappears for each selected job. For example, if you select five jobs, the dialog will appear five times.
A valid dependency sidecar file is usually required. For more details, see the section on Dependencies
After a few minutes, new Workers pop-up in the Worker panel with the suffix -conductor. Each Worker will render exactly one Deadline task. After a Worker completes a Deadline task, they will shut down.
To delete Conductor Workers that have completed their task, visit the Workers panel. They will have a state of Stalled.

How it works¶

(none-native mode)

We run a Deadline Worker job on Conductor and configure it to shut down after exactly one Deadline task. There's some additional logic to create a temporary Deadline group (conductorautogroup_<jobid>) to ensure that the submitted Deadline job only runs on the Conductor instances and that the Conductor instances only pick up the given Deadline job.

An essential step is to ensure that a valid sidecar dependency file exists. If it doesn't, the Conductor instances will not have any necessary files to run the task. See the section on dependencies for more details.

Viewing reports and most other functions of a Deadline Worker will work with a Conductor Deadline Worker, except for any operation that involves sending a command directly to the host instance (see below for more details).

Dependencies¶

A task can not render if it does not have all of its required dependencies. We submit a list of all the required dependencies to Conductor as part of the submission process. The uploader daemon is responsible for performing the upload of the data. Therefore, it's necessary to create this list of dependencies beforehand.

Note

For convenience, a sidecar file is not required if the only file needed for the job is the selected scene file

There are several options to create the list of dependencies:

Generate the sidecar dependency file on job submission. Do this manually or as part of the submission to Deadline. You'll find a modified Deadline Maya submitter included in the repo (this should only be used as a sample). Below is a python code snippet on how to scan and save dependencies:

This code has been depreciated as it uses our old client_tools library. Please take a look at ciomaya for updated method and/or reach out to Conductor's support team

def get_dependencies():
    from conductor.lib import common, maya_utils
    resources = common.load_resources_file()
    dependency_attrs = resources.get("maya_dependency_attrs") or {}
    return maya_utils.collect_dependencies(dependency_attrs)

dependencies = {'dependencies': get_dependencies()}

ma_path = maya.cmds.file(query=True, sceneName=True)
dependency_sidecar_path = "{}.cdepends".format(ma_path)

with open(dependency_sidecar_path, 'w') as fh:
    json.dump(dependencies, fh)

Submit an upstream job to Deadline that will generate the dependency sidecar file

Generally, if a DCC software is already open with the scene file loaded, scanning for dependencies is a quick task.

Note

There are a few key differences between regular on-site Deadline Workers and Conductor Workers:

Conductor Workers are not directly accessible like other Workers. They run from within the Conductor infrastructure. They don't have an exposed IP/hostname, and therefore, remote commands do not work.
Conductor Workers can only render the jobs they're assigned. If you try to render a different job, it will fail (as none of the files are available).
Conductor Workers will manage themselves. They'll start-up and shut down as needed

Package Mapping¶

Conductor also needs to know what software is required for the job. This can be obtained from the Deadline job. The DeadlineToConductorPackageMapper class is used for this purpose and will likely need to be modified by the customer to support the customer's specific DCC's and plugins. More details on extending DeadlineToConductorPackageMapper are available in the code-level documentation. Please contact Conductor support with any questions.

Licensing¶

Deadline 10.1.23 and greater no longer require a license. Earlier versions of 10.1 don't require a license if running on AWS. It is the customer's responsibility to choose the appropriate licensing model. Licensing of DCC's and all other software is managed by Conductor and included in the cost.

Licenses can be assigned to Conductor Deadline Workers by using the Auto Configure feature in the Repository options.

Supported software¶

Currently, the Deadline integration is set up to support Maya with Arnold, VRay and Renderman renderers using the MayaCmd plugin. Arnold standalone and generic Cmd plugins are also supported. Additional software can be supported by extending conductor_deadline.plugin_mapper.DeadlinePluginMapper.

It is possible to upload the files simultaneously as submitting the job without using the uploader daemon. This can be configured in the Conductor config. However, this method is not recommended when submitting jobs via Deadline due to the potentially long wait times.* ↩