Basics

Overview

This page outlines the basic Conducto features. It aims to capture the big picture, without dwelling too much on details. If you're more interested in a deep-dive into something in particular, follow a link below or check out the subsections in the sidebar.

Pipeline Structure

A Conducto pipeline is a tree which controls how commands are executed. Exec type nodes are at the leaves of the tree--they run shell commands in containers. Parallel and Serial nodes control the way that their children run. Conducto provides an API so you can arrange these nodes in code to make a pipeline definition.

Controlling a Pipeline

If you have a definition, you can use it to create a pipeline instance. For the new instance, you'll need to indicate whether you want the Exec nodes to create containers on your local machine (which is free), or in the cloud. Either way, you can interact with the pipeline at conducto.com/app. From there you can run the pipeline, examine results, make changes, and rerun all or part of the pipeline. The history of each node execution is saved with the pipeline, so you can see the impact of any changes you made.

Images

When an Exec node runs a command, it creates a container for that command. The container's contents may change based on your command, but it's based on an initial disk image, which remains fixed. You can control which software is available to the container by making modifications to the image before the container gets created. If you make changes, you'll want to be aware of whether you're changing something in the image, or something in the container, because it will effect how much reprocessing is required before you can see your changes in action.

Debugging

If you arrange your pipeline well, the Conducto web app will quickly draw your attention to problem areas. But it can only zoom in so closely--at some point you're going to need tools that are specifically suited for that node. To do this, consult the pipeline via the web app and copy a debug command.

Pasting it into your terminal will give you an interactive shell session in a container that's just abount to run the suspicious command, so you can poke around, attach, a debugger, or work whatever magic you need. When you're ready to catch that bug, you can run the same script that Conducto would run, and expect the same behavior that you'd see in-pipeline.

Native Functions

If you're writing your pipeline definition in python, and you want your node to call a python function, there's a shortcut. Rather that setting up your Exec nodes to run shell commands that invoke python you can just initialize the Exec node with the function that you want it to call.

Next Steps

The functionality covered so far should get you pretty far. If you're looking for more, consider the Advanced Section.