sevenbridges-python documentation!¶

Import the library¶

You should begin by importing the API library sevenbridges-python to your python script that will interact with the API:

import sevenbridges as sbg

Then, use one of the following three methods to initialize the library:

api = sbg.Api(url='https://api.sbgenomics.com/v2', token='<TOKEN_HERE>')

import os

# Usually these variables would be set in the shell beforehand
os.environ['SB_API_ENDPOINT'] = '<https://api.sbgenomics.com/v2 or https://cgc-api.sbgenomics.com/v2>'
os.environ['SB_AUTH_TOKEN'] = '<TOKEN_HERE>'

api = sbg.Api()

[default]
api_endpoint = https://api.sbgenomics.com/v2
auth_token = <TOKEN_HERE>

[cgc]
api_endpoint = https://cgc-api.sbgenomics.com/v2
auth_token = <TOKEN_HERE>

c = sbg.Config(profile='cgc')
api = sbg.Api(config=c)

List Projects - introduction to pagination and iteration¶

In order to list your projects, invoke the api.projects.query method. This method follows server pagination and therefore allows pagination parameters to be passed to it. Passing a pagination parameter controls which resources you are shown. The offset parameter controls the start of the pagination while the limit parameter controls the number of items to be retrieved.

Below is an example of how to get all your projects, using the query method and the pagination parameters offset of 0 and limit of 10.

project_list = api.projects.query(offset=0, limit=10)

project_list has now been defined to be an object of the type collection which acts just like a regular python list, and so supports indexing, slicing, iterating and other list functions. All collections in the sevenbridges-python library have two methods: next_page and previous_page which allow you to load the next or previous pagination pages.

There are several things you can do with a collection of any kind of object:

1. The generic query, e.g. api.projects.query(), accepts the pagination parameters offset and limit as introduced above.
2. If you wish to iterate on a complete collection use the all() method, which returns an iterator
3. If you want to manually iterate on the collection (page by page), use next_page() and previous_page() methods on the collection.
4. You can easily cast the collection to the list, so you can re-use it later by issuing the standard Python project_list = list(api.projects.query().all()).

# Get details of my first 10 projects.
project_list = api.projects.query(limit=10)

# Iterate through all my projects and print their name and id
for project in api.projects.query().all():
    print (project.id,project.name)

# Get all my current projects and store them in a list
my_projects = list(api.projects.query().all())

Get details of a single project¶

You can get details of a single project by issuing the api.projects.get() method with the parameter id set to the id of the project in question. Note that this call, as well as other calls to the API server may raise an exception which you can catch and process if required.

Note - To process errors from the library, import SbgError from sevenbridges.errors, as shown below.

from sevenbridges.errors import SbgError
try:
    project_id = 'doesnotexist/forsure'
    project = api.projects.get(id=project_id)
except SbgError as e:
    print (e.message)

Errors in SbgError have the properties code and message which refer to the number and text of 4-digit API status codes that are specific to the Seven Bridges Platform and API. To see all the available codes, see the documentation:

• http://docs.sevenbridges.com/docs/api-status-codes for the Seven Bridges Platform
• http://docs.cancergenomicscloud.org/docs/api-status-codes for the CGC.

Project properties¶

Once you have obtained the id of a Project instance, you can see its properties. All projects have the following properties:

href - Project href on the API

id - Id of the project

name - name of the project

description - description of the project

billing_group - billing group attached to the project

type - type of the project (v1 or v2)

tags - list of project tags

The property href href is a URL on the server that uniquely identifies the resource in question. All resources have this attribute. Each project also has a name, identifier, description indicating its use, a type, some tags and also a billing_group identifier representing the billing group that is attached to the project.

Project methods – an introduction to methods in the sevenbridges-python library¶

There are two types of methods in the sevenbridges-python library: static and dynamic. Static methods are invoked on the Api object instance. Dynamic methods are invoked from the instance of the object representing the resource (e.g. the project).

Static methods include:

1. Create a new resource: for example, api.projects.create(name="My new project", billing_group='296a98a9-424c-43f3-aec5-306e0e41c799') creates a new resource. The parameters used will depend on the resource in question.
2. Get a resource: the method api.projects.get(id='user/project') returns details of a specific resource, denoted by its id.
3. Query resources - the method api.projects.query() method returns a pageable list of type collection of projects. The same goes for other resources, so api.tasks.query(status='COMPLETED') returns a collection of completed tasks with default paging.

Dynamic methods can be generic (for all resources) or specific to a single resource. They are called on a concrete object, such as a Project object.

So, suppose that project is an instance of Project object. Then, we can:

1. Delete the resource: project.delete() deletes the object (if deletion of this resource is supported on the API).
2. Reload the resource from server: project.reload() reloads the state of the object from the server.
3. Save changes to the server: project.save() saves all properties

The following example shows some of the methods used to manipulatate projects.

# Get a collection of projects
projects = api.projects.query()

# Grab the first billing group
bg = api.billing_groups.query(limit=1)[0]

# Create a project using the billing group grabbed above
new_project = api.projects.create(name="My new project", billing_group=bg.id)

# Add a new member to the project
new_project.add_member(user='newuser', permissions= {'write':True, 'execute':True})

Other project methods include:

1. Get members of the project and their permissions - project.get_members() - returns a Collection of members and their permissions
2. Add a member to the project - project.add_member()
3. Remove a member from the project - project.remove_member()
4. List files from the project - project.get_files()
5. Add files to the project - project.add_files() - you can add a single File or a Collection of files
6. List apps from the project - project.get_apps()
7. List tasks from the project - project.get_tasks()

Manage billing groups¶

Querying billing groups will return a standard collection object.

# Query billing groups
bgroup_list = api.billing_groups.query(offset=0, limit=10)

# Fetch a billing group's information
bg = api.billing_groups.get(id='f1969c90-da54-4118-8e96-c3f0b49a163d')

Billing group properties¶

The following properties are attached to each billing group:

href - Billing group href on the API server.

id - Billing group identifier.

owner - Username of the user that owns the billing group.

name - Billing group name.

type - Billing group type (free or regular)

pending - True if billing group is not yet approved, False if the billing group has been approved.

disabled - True if billing group is disabled, False if its enabled.

balance - Billing group balance.

Billing group methods¶

There is one billing group method:

breakdown() fetches a cost breakdown by project and analysis for the selected billing group.

Manage invoices¶

Querying invoices will return an Invoices collection object.

invoices = api.invoices.query()

Once you have obtained the invoice identifier you can also fetch specific invoice information.

invoices = api.invoices.get(id='6351830069')

Invoice properties¶

The following properties are attached to each invoice.

href - Invoice href on the API server.

id - Invoice identifier.

pending - Set to True if invoice has not yet been approved by Seven Bridges, False otherwise.

analysis_costs - Costs of your analysis.

storage_costs - Storage costs.

total - Total costs.

invoice_period - Invoicing period (from-to)

File properties¶

Each file has the following properties:

href - File href on the API server.

id - File identifier.

name - File name.

size - File size in bytes.

project - Identifier of the project that file is located in.

created_on - Date of the file creation.

modified_on - Last modification of the file.

origin - File origin information, indicating the task that created the file.

tags - File tags.

metadata - File metadata.

File methods¶

Files have the following methods:

• Refresh the file with data from the server: reload()
• Copy the file from one project to another: copy()
• Download the file: download()
• Save modifications to the file to the server save()
• Delete the resource: delete()

See the examples below for information on the arguments these methods take:

Examples¶

# Filter files by name to find only file names containing the specified string:
files = api.files.query(project='user/my-project')
my_file = [file for file in files if 'fasta' in file.name]

# Or simply query files by name if you know their exact file name(s)
files = api.files.query(project='user/myproject', names=['SRR062634.filt.fastq.gz','SRR062635.filt.fastq.gz'])
my_files = api.files.query(project='user/myproject', metadata = {'sample_id': 'SRR062634'} )


# Edit a file's metadata
my_file = my_files[0]
my_file.metadata['sample_id'] = 'my-sample'
my_file.metadata['library'] = 'my-library'


# Add metadata (if you are starting with a file without metadata)
my_file = my_files[0]
my_file.metadata = {'sample_id' : 'my-sample',
                    'library' : 'my-library'
                  }

# Also set a tag on that file
my_file.tags = ['example']

# Save modifications
my_file.save()

# Copy a file between projects
new_file = my_file.copy(project='user/my-other-project', name='my-new-file')

# Download a file to the current working directory
# Optionally, path can contain a full path on local filesystem
new_file.download(path='my_new_file_on_disk')

File Download¶

Synchronous file download:

file = api.files.get('file-identifier')
file.download('/home/bar/foo/file.bam')

Asynchronous file download:

file = api.files.get('file-identifier')
download = file.download('/home/bar/foo.bam', wait=False)

download.path # Gets the target file path of the download.
download.status # Gets the status of the download.
download.progress # Gets the progress of the download as percentage.
download.start_time # Gets the start time of the download.
download.duration # Gets the download elapsed time.

download.start() # Starts the download.
download.pause() # Pauses the download.
download.resume() # Resumes the download.
download.stop() # Stops the download.
download.wait() # Block the main loop until download completes.

You can register the callback or error callback function to the download handle: download.add_callback(callback=my_callback, errorback=my_error_back)

Registered callback method will be invoked on completion of the download. The errorback method will be invoked if error happens during download.

File Upload¶

Synchronous file upload:

# Get the project where we want to upload files.
project = api.projects.get('project-identifier')
api.files.upload('/home/bar/foo/file.fastq', project)
# Optionally we can set file name of the uploaded file.
api.files.upload('/home/bar/foo/file.fastq', project, file_name='new.fastq')

Asynchronous file upload:

upload = api.files.upload('/home/bar/foo/file.fastq', 'project-identifier', wait=False)

upload.file_name # Gets the file name of the upload.
upload.status # Gets the status of the upload.
upload.progress # Gets the progress of the upload as percentage.
upload.start_time # Gets the start time of the upload.
upload.duration # Gets the upload elapsed time.

upload.start() # Starts the upload.
upload.pause() # Pauses the upload.
upload.resume() # Resumes the upload.
upload.stop() # Stops the upload.
upload.wait() # Block the main loop until upload completes.

You can register the callback or error callback in the same manner as it was described for asynchronous file download.

Volume properties¶

Each volume has the following properties:

href - Volume href on the API server.

id - Volume identifier in format owner/name.

name - Volume name. Learn more about this in our Knowledge Center.

access_mode - Whether the volume was created as read-only (RO) or read-write (RW). Learn more about this in our Knowledge Center.

active - Whether or not this volume is active.

created_on - Time when the volume was created.

modified_on - Time when the volume was last modified.

description - An optional description of this volume.

service - This object contains the information about the cloud service that this volume represents.

Volume methods¶

Volumes have the following methods:

• Refresh the volume with data from the server: reload()
• Get imports for a particular volume get_imports()
• Get exports for a particular volume get_exports()
• Create a new volume based on the AWS S3 service - create_s3_volume()
• Create a new volume based on Google Cloud Storage service - create_google_volume()
• Save modifications to the volume to the server save()
• Unlink the volume delete()

See the examples below for information on the arguments these methods take:

Examples¶

# Create a new volume based on AWS S3 for importing files
volume_import = api.volumes.create_s3_volume(name='my_input_volume', bucket='my_bucket',access_key_id='AKIAIOSFODNN7EXAMPLE',secret_access_key = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',access_mode='RO')

# Create a new volume based on AWS S3 for exporting files
volume_export = api.volumes.create_s3_volume(name='my_output_volume', bucket='my_bucket', access_key_id='AKIAIOSFODNN7EXAMPLE',secret_access_key = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',access_mode='RW')
# List all volumes available
volumes = api.volumes.query()

Import properties¶

When you import a file from a volume into a project on the Platform, you are importing a file from your cloud storage provider (Amazon Web Services or Google Cloud Storage) via the volume onto the Platform.

If successful, an alias will be created on the Platform. Aliases appear as files on the Platform and can be copied, executed, and modified as such. They refer back to the respective file on the given volume.

Each import has the following properties:

href - Import href on the API server.

id - Import identifier.

source - Source of the import, object of type VolumeFile, contains info on volume and file location on the volume

destination - Destination of the import, object of type ImportDestination, containing info on project where the file was imported to and name of the file in the project

state - State of the import. Can be PENDING, RUNNING, COMPLETED and FAILED.

result - If the import was completed, contains the result of the import - a File object.

error - Contains the Error object if the import failed.

overwrite - Whether the import was set to overwrite file at destination or not.

started_on - Contains the date and time when the import was started.

finished_on - Contains the date and time when the import was finished.

Import methods¶

Imports have the following methods:

• Refresh the import with data from the server: reload()
• Start an import by specifying the source and the destination of the import - submit_import()
• Delete the import - delete()

See the examples below for information on the arguments these methods take:

Examples¶

# Import a  file to a project
my_project = api.projects.get(id='my_project')
bucket_location = 'fastq/my_file.fastq'
imp = api.imports.submit_import(volume=volume_import, project=my_project, location=bucket_location)
# Wait until the import finishes
while True:
      import_status = imp.reload().state
      if import_status in (ImportExportState.COMPLETED, ImportExportState.FAILED):
           break
      time.sleep(10)
# Continue with the import
if imp.state == ImportExportState.COMPLETED:
      imported_file = imp.result

Export properties¶

When you export a file from a project on the Platform into a volume, you are essentially writing to your cloud storage bucket on Amazon Web Services or Google Cloud Storage via the volume.

Note that the file selected for export must not be a public file or an alias. Aliases are objects stored in your cloud storage bucket which have been made available on the Platform.

The volume you are exporting to must be configured for read-write access. To do this, set the access_mode parameter to RW when creating or modifying a volume. Learn more about this from our Knowledge Center.

Each export has the following properties:

href - Export href on the API server.

id - Export identifier.

source - Source of the export, object of type File

destination - Destination of the export, object of type VolumeFile, containing info on project where the file was imported to and name of the file in the project

state - State of the export. Can be PENDING, RUNNING, COMPLETED and FAILED.

result - If the export was completed, this contains the result of the import - a File object.

error - Contains the Error object if the export failed.

overwrite - Whether or not the export was set to overwrite the file at the destination.

started_on - Contains the date and time when the export was started.

finished_on - Contains the date and time when the export was finished.

Export methods¶

Exports have the following methods:

• Refresh the export with data from the server: reload()
• Submit export, by specifying source and destination of the import: submit_import()
• Delete the export: delete()

See the examples below for information on the arguments these methods take:

Examples¶

# Export a set of files to a volume
# Get files from a project
files_to_export = api.files.query(project=my_project).all()
# And export all the files to the output bucket
exports = []
for f in files_to_export:
      export = api.exports.submit_export(file=f, volume = volume_export, location=f.name)
      exports.append(export)
# Wait for exports to finish:
num_exports = len(exports)
done = False

while not done:
      done_len = 0
      for e in exports:
             if e.reload().state in (ImportExportState.COMPLETED, ImportExportState.FAILED):
                    done_len += 1
             time.sleep(10)
      if done_len == num_exports:
             done = True

App properties¶

Each app has the following available properties:

href - The URL of the app on the API server.

id - App identifier.

name - App name.

project - Identifier of the project that app is located in.

revision - App revision.

raw - Raw CWL description of the app.

App methods¶

• App only has class methods that were mentioned above.

Task properties¶

href - Task URL on the API server.

id - Task identifier.

name - Task name.

status - Task status.

project - Identifier of the project that the task is located in.

app - The identifier of the app that was used for the task.

type - Task type.

created_by - Username of the task creator.

executed_by- Username of the task executor.

batch - Boolean flag: True for batch tasks, False for regular & child tasks.

batch_by - Batching criteria.

batch_group - Batch group assigned to the child task calculated from the batch_by criteria.

batch_input - Input identifier on to which to apply batching.

parent - Parent task for a batch child.

end_time - Task end time.

execution_status - Task execution status.

price - Task cost.

inputs - Inputs that were subbmited to the task.

outputs - Generated outputs from the task.

Task methods¶

The following class and instance methods are available for tasks:

• Create a task on the server and, optionally, run it: create().
• Query tasks: query().
• Get single task’s information: get().
• Abort a running task: abort().
• Run a draft task: run()
• Delete a draft task from the server: delete().
• Refresh the task object information with the date from the server: refresh().
• Save task modifications to the sever: save().
• Get task exection datails: get_execution_details().
• Get batch children if the task is a batch task: get_batch_children().

Task creation hints¶

• Both input files and parameters are passed the same way together in a single dictionary to inputs.
• api.files.query always return an array of files. For single file inputs, use api.files.query(project='my-project', names=["one_file.fa"])[0].

Task Examples¶

Single task¶

# Task name
task_name = 'my-first-task'

# Project in which I want to run a task.
project_id = 'my-username/my-project'

# App I want to use to run a task
app = 'my-username/my-project/my-app'

# Inputs
inputs = {}
inputs['FastQC-Reads'] = api.files.query(project='my-project', metadata={'sample': 'some-sample'})

try:
    task = api.tasks.create(name=name, project=project, app=app, inputs=inputs, run=True)
except SbError:
    print('I was unable to run the task.')

# Task can also be ran by invoking .run() method on the draft task.
task.run()

Batch task¶

# Task name
task_name = 'my-first-task'

# Project in which to run the task.
project_id = 'my-username/my-project'

# App to use to run the task
app = 'my-username/my-project/my-app'

# Inputs
inputs = {}
inputs['FastQC-Reads'] = api.files.query(project='my-project', metadata={'sample': 'some-sample'})

# Specify that one task should be created per file (i.e. batch tasks by file).
batch_by = {'type': 'item'}


# Specify that the batch input is FastQC-Reads
batch_input = 'FastQC-Reads'

try:
    task = api.tasks.create(name=name, project=project, app=app,
                            inputs=inputs, batch_input=batch_input, batch_by=batch_by, run=True)
except SbError:
    print('I was unable to run a batch task.')