Configuration File

Overview

This is the blueprint that defines an applications CI/CD workflow, container configuration, container host configuration, and its dependencies per environment and data-center. The file should be written in yaml (json is also supported). The objects in the configuration file have a cascading effect. Meaning that the objects defined deepest in the object structure take precedent over previously defined values. This allows for common configuration values to be shared as well as allowing the flexibility to override values per individual operation.

Warning

Configuration injection isn’t included in this configuration file.

Terminology Explanation

Definitions:

freight-forwarder.yml	Handles the organization of application services, environments and data-centers.
hosts	A server physical/virtual server that has the docker daemon running. The daemon must be configured to communicate via tcp.
service	Multiple services are defined to make a project. A service could be an api, proxy, db, etc.
environments	An object that is used to define where/how containers and images are being deployed, exported, and tested for each environment.
data centers	An object that is used to define where/how containers and images are being deployed, exported, and tested for each data center.
registry	The docker registry where images will be pushed.

Root Level Properties

All of the properties at the root of the configuration file either correlate to a service, project metadata, or environments.

Name	Required	Type	Description
team	True	string	Name of the development team.
project	True	string	The project that is being worked on.
repository	True	string	The projects git repo.
services	True	object	Refer to Service Properties
registries	False	object	Refer to Registries Properties
environments	True	object	Refer to Environments properties

---
# team name
team: "itops"

# current project
project: "cia"

# git repo
repository: "git@github.com:Adapp/cia.git"

# Service definition
api:
  build: "./"
  test: "./tests/"
  ports:
      - "8000:8000"

# environments object is collection of environments and data centers
environments:

  # development environment
  development:

    # local datacenter
    local:

    # sea1 data center
    sea1:

  # staging environment
  staging:

  # production environment
  production:

Service Properties

Each service object is a component that defines a specific service of a project. An example would be an api or database. Services can be built from a Dockerfile or pulled from an image in a docker registry. The container and host configuration can be modified on a per service bases.

Name	Required	Type	Description
build	one of	string	Path to the service Dockerfile.
test	False	string	Path to a test Dockerfile that should be used to verify images before pushing them to a docker registry.
image	one of	string	The name of the docker image in which the service depends on. If its being pulled from a registry the fqdn must be provided. Example: registry/itops/cia:latest. If the image property is spectified it will always take precedent over the build property. If a service object has both an image and build specified the image will exclusively be used.
export_to	False	string	Registry alias where images will be push. This will be set to the default value if nothing is provided. The alias is defined in Registries Properties
Container Config	any of		Refer to Container Config Properties

---
# service alias.
couchdb:

  # Docker image to use for service.
  image: "registry_alias/itops/cia-couchdb:local-development-latest"

  # Path to Dockerfile.
  build: ./docker/couchdb/Dockerfile

  # Synonyms with -d from the docker cli.
  detach: true

  # Synonyms with -p from the docker cli.
  ports:
    - "6984:6984"
    - "5984:5984"

Registries Properties

The registries object is a grouping of docker registries that images will be pulled from or pushed to. The alias of each registry can be used in any image definition image: docker_hub/library/centos:6.6. By default docker_hub is provided for all users. The default property will be set to docker_hub unless overridden with any of the defined registries.

Name	Required	Type	Description
registry (alias)	True	object	Refer to Registry Properties
default	False	object	Refer to Registry Properties

# define
registries:
  # define development registry
  tune_development: &default_registry
    address: "https://example_docker_registry"
    verify: false

  # define production registry
  tune_production:
    address: "https://o-pregister-sea1.ops.tune.com"
    ssl_cert_path: /path/to/certs
    verify: false

    # define auth for production registry
    auth:
        type: "registry_rubber"
        address: "https://o-regrubber-sea1.ops.tune.com"
        ssl_cert_path: /path/to/certs
        verify: false

  # define default registry. If this isn't defined default will be docker_hub.
  default: *default_registry

Registry Properties

The docker registry that will be used to pull or push validated docker images.

Name	Required	Type	Description
address	True	string	Address of docker host, must provide http scheme. Example: https://your_dev_box.office.priv:2376
ssl_cert_path	False	string	Full system path to client certs. Example: /etc/docker/certs/client/dev/
verify	False	bool	Validate certificate authority?
auth	False	object	Refer to Registry Auth Properties

---
registries:
  # registry definition
  default:
    address: "https://docker-dev.ops.tune.com"
    verify: false

Registry Auth Properties

These are properties required for authentication with a registry. Currently basic and registry_rubber auth are support. Dynamic auth uses Registry Rubber to support nonce like basic auth credentials. Please refer to Registry Rubber documentation for a deeper understanding of the service.

Name	Required	Type	Description
address	True	string	Address of docker host, must provide http scheme. Example: https://your_dev_box.office.priv:2376
ssl_cert_path	False	string	Full system path to client certs. Example: /etc/docker/certs/client/dev/
verify	False	bool	Validate certificate authority?
type	False	string	Type of auth. Currently supports basic and registry_rubber. Will default to basic.

---
registries:
  # registry definition
  default:
    address: "https://example-docker-registry.com"
    ssl_cert_path: /path/to/certs
    verify: false

    # optional: required if using registry rubber.
    auth:
      type: "registry_rubber"
      address: "https://o-regrubber-sea1.ops.tune.com"
      ssl_cert_path: "/etc/docker/certs/registry/build"
      verify: false

Environments properties

The Environments object is a grouping of instructions and configuration values that define the behavior for a CI/CD pipeline based on environment and data center. The environments and data centers are both user defined.

Warning

If using CIA: The environments and data centers need to match what is defined in CIA. Freight Forwarder will pass these values to the injector to obtain the correct configuration data.

Name	Required	Type	Description
environment	True	object	Refer to Environment Properties valid environments are ci, dev, development, test, testing, perf, performance, stage, staging, integration, prod, production.
service	False	object	Refer to Service Properties
host	False	object	Refer to Hosts Properties

---
# environments object definition
environments:
  # define a host as a variable to use later
  boot2docker: &boot2docker
    - address: "https://192.168.99.100:2376"
      ssl_cert_path: /path/to/certs
      verify: false

  # override api service APP_ENV environment variable.
  api:
    env_vars:
      - "APP_ENV=development"

  # define development environment
  development:

    # define local datacenter for development
    local:
      hosts:
        default: *boot2docker

  # define staging environment
  staging:

    # define staging datacenter in sea1
    sea1: {}

  # define production environment
  production:

    # define us-east-01 for production datacenter.
    us-east-01: {}

Environment Properties

The environment of the application. An application can and one or many environments. Valid environments are ci, dev, development, test, testing, perf, performance, stage, staging, integration, prod, production.

Name	Required	Type	Description
hosts	False	object	Refer to Hosts Properties if not defined freight forwarder will use the docker environment variables.
data centers	True	object	Refer to Data Center Properties
services	False	object	Refer to Service Properties

---
environments:

  # define development environment
  development:

    # define local datacenter.
    local:

      # define development local hosts.
      hosts:

        # define default hosts
        default:
          - address: "https://192.168.99.100:2376"
            ssl_cert_path: /path/to/certs
            verify: false

      # override api service APP_ENV environment variable for development local.
      api:
        env_vars:
          - "APP_ENV=development"

  # define production environment
  production:

    # define production hosts
    hosts:
      # define hosts specificly for the api service.
      api:
        - address: "https://192.168.99.102:2376"
          ssl_cert_path: /path/to/certs
          verify: false

      # define default hosts
      default:
        - address: "https://192.168.99.101:2376"
          ssl_cert_path: /path/to/certs
          verify: false

    # override api service APP_ENV environment variable for production.
    api:
      env_vars:
        - "APP_ENV=production"

Data Center Properties

Each environment can have multiple data center objects. Some examples of data centers: local, sea1, use-east-01, and us-west-02

Name	Required	Type	Description
hosts	False	object	Refer to Hosts Properties if not defined freight forwarder will use the docker environment variables.
service	False	object	Refer to Service Properties
deploy	one of	object	Refer to Deploy Properties
export	one of	object	Refer to Export Properties
quality_control	one of	object	Refer to Quality Control Properties

---
environments:

  # define development environment.
  development:

    # define local datacenter.
    local:

      # define hosts for development local.
      hosts:

        # define default hosts.
        default:
          - address: "https://192.168.99.100:2376"
            sslCertPath: "/Users/alexb/.docker/machine/machines/ff01-dev"
            verify: false

         # define host to use during export
         export:
           - address: "https://your-ci-server.sea1.office.priv:2376"
             sslCertPath: "/path/to/your/certs/"
             verify: false

      # define deploy command orderides
      deploy:

        # override ui service properties.
        ui:
          image: registry_alias/itops/cia-ui:local-development-latest
          volumes:
            - /var/tune/cia-ui/public/

        # override static-assets service properties.
        static-assets:
          image: registry_alias/itops/cia-static-assets:local-development-latest
          volumes:
            -  /static/
          volumes_from: []

      export:
        ui:
          export_to: registry_alias

        static-assets:
          export_to: registry_alias

Deploy Properties

The deploy object allows development teams to define unique deployment behavior for specific service, environment, and data center.

Name	Required	Type	Description
service	True	object	Refer to Service Properties

---
registries:

  registry_alias: &registry_alias
    address: https://docker-registry-example.com
    ssl_cert_path: /path/to/certs
    verify: false

  default: *registry_alias

environments:
  production:
    # define datacenter
    us-west-02:

      # define deploy action
      deploy:

        # deployment overrides for static-assets
        static-assets:
          image: registry_alias/itops/cia-static-assets:latest
          volumes:
            - /static/
          volumes_from: []
          restart_policy: null

        # deployment overrides for api
        api:
          image: registry_alias/itops/cia-api:o-ciapi03-2b-production-latest

        # deployment overrides for nginx
        nginx:
          image: registry_alias/itops/cia-nginx:us-west-02-production-latest

Export Properties

The export object allows development teams to define unique artifact creation behavior for a specific service, environment, and data center. Export is the only action that allows you to have a specific unique hosts definition (this is a good place for a jenkins or build host).

Note

To remove Freight Forwarders tagging scheme pass –no-tagging-scheme to the cli export command.

Warning

When exporting images Freight Forwarder will use the service definition in deploy for any dependencies/dependents. In addition, if a command is provided in the config for the service being exported Freight Forwarder assumes any changes made should be committed into the image.

Name	Required	Type	Description
service	True	object	Refer to Service Properties
tags	False	array[string]	A list of tags that should be applied to the image before exporting.

---
# environments
environments:
  production:
  # datacenter definition
    us-west-02:
      # hosts for us-west-02
      hosts:

        # default hosts
        default:
          - address: "https://dev_hostname:2376"
            ssl_cert_path: /path/to/certs
            verify: false

        # host specific to the export action. will default to hosts defined in
        # default if not provided.
        export:
          - address: "https://127.0.0.1:2376"
            ssl_cert_path: /path/to/certs
            verify: false

      # overrides for the export action.
      export:

        # api service export specific overrides.
        api:
          env_vars:
            - APP_ENV=production

          # specify what registry to export to.
          export_to: registry_alias

Quality Control Properties

The quality control object allows developers a way to test containers, images, and workflows locally before deploying or exporting.

Name	Required	Type	Description
service	True	object	Refer to Service Properties

---
# quality control action.
quality_control:

  # couchdb service overrides.
  couchdb:
    log_config:
      type: json-file
      config: {}

    ports:
      - "6984:6984"

  # api service overrides.
  api:
    links: []
    env_vars:
      - APP_ENV=development

Hosts Properties

The hosts object is a collection of docker hosts in which Freight Forwarder will interact with when deploying, exporting, or testing. Each service can have a collection of its own hosts but will default to the defaults definition or the standard Docker environment variables: DOCKER_HOST, DOCKER_TLS_VERIFY, DOCKER_CERT_PATH.

Name	Required	Type	Description
service_name (alias)	one of	list[Host Properties]	List of Host Properties
export	one of	list[Host Properties]	List with as single element of Host Properties
default	one of	list[Host Properties]	List of Host Properties

---
# development environment definition.
development:
  # development environment local datacenter definition.
  local:

    # hosts definition.
    hosts:

      # default hosts.
      default:
        - address: "https://192.168.99.100:2376"
          ssl_cert_path: /path/to/certs
          verify: false
        - address: "https://192.168.99.110:2376"
          ssl_cert_path: /path/to/certs
          verify: false

Host Properties

The host object is metadata pertaining to docker hosts. If using ssl certs they must be the host where Freight Forwarder is run and be able to be read by the user running the commands.

Name	Required	Type	Description
address	True	string	Address of docker host, must provide http scheme. Example: https://your_dev_box.office.priv:2376
ssl_cert_path	False	string	Full system path to client certs. Example: /etc/docker/certs/client/dev/
verify	False	bool	Validate certificate authority?

---
# development environment definition.
development:
  # development environment local datacenter definition.
  local:

    # hosts definition.
    hosts:

      # default hosts.
      default:
        - address: "https://192.168.99.100:2376"
          ssl_cert_path: /path/to/certs
          verify: false

      # host to build and export from.
      export:
        - address: "https://192.168.99.120:2376"
          ssl_cert_path: /path/to/certs
          verify: false

      # specific hosts for the api service.
      api:
        - address: "https://192.168.99.110:2376"
          ssl_cert_path: /path/to/certs
          verify: false

Host Config Properties

Host configuration properties can be included as a part of the the service definition. This allows for greater control when configuring a container for specific requirements to operate. It is suggested that a root level definition of a service be minimalistic compared to how it should be deployed in a specific environment or data-center.

Refer to Docker Docs for the full list of of potential properties.

Name	Required	Type	Default Value	Description
binds	False	list	[‘/dev/log:/dev/log:rw’]	Default value applied to all containers. This allows for inherit use of /dev/log for logging by the container
cap_add	False	string	None	Defined system capabilities to add to the container from the host. Refer to http://linux.die.net/man/7/capabilities for a full list of capabilities
cap_drop	False	string	None	Defined system capabilities to remove from the container from the host. Refer to http://linux.die.net/man/7/capabilities for a full list of capabilities
devices	False	list	None	Device to add to the container from the host Format of devices should match as shown below. Permissions need to be set appropriately. “/path/to/dev:/path/inside/container:rwm’
links	False	list	[]	Add link to another container
lxc_conf	False	list	[]	Add custom lxc options
readonly_root_fs	False	boolean	False	Read-only root filesystem
readonly_rootfs
security_opt	False	list	None	Security Options
memory	False	int	0	Memory limit
memory_swap	False	int	0	Total memory (memory + swap), -1 to disable swap
cpu_shares	False	int	0	CPU shares (relative weight)
port_bindings	False	list/dict	{}	Map the exposed ports from host to the container
ports
publish_all_ports	False	boolean	False	All exposed ports are associated with an ephemeral port
privileged	False	boolean	False	Give extended privileges to this container
dns	False		None	Set custom DNS servers
dns_search	False		None	Set custom DNS search domains
extra_hosts	False		None	Add additional hosts as needed to the container
network_mode	False	string	bridge	Network configuration for container environment
volumes_from	False	list	[]	Mount volumes from the specified container(s)
cgroup_parent	False	string	‘’	Optional parent cgroup for the container
log_config	False	dict	json-file	Defined logging configuration for the container. Reference the logging-driver for appropriate docker engine version Default Value: { “type”: json-file, “config”: { “max-files”: “2”, “max-size”: “100m” } }
ulimits	False	dict	None	Defined user process resource limits for the containers run time environment
restart_policy	False	dict	{}	This defines the behavior of the container on failure

Container Config Properties

Container config properties are container configuration settings that can be changed by the developer to meet the container run time requirements. These properties can be set at any level but the furthest in the object chain will take presidents. Please refer to Docker Docs for a full list of properties.

Name	Required	Type	Default Value	Description
attach_stderr	False	boolean	False	Attach to stderr
attach_stdin	False	boolean	False	Attach to stdin and pass input into Container
attach_stdout	False	boolean	False	Attach to stdout
cmd	False	list	None	Override the command directive on the container
command
domain_name	False	string	‘’	Domain name for the container
domainname
entry_point	False	list	‘’	Defined entrypoint for the container
entrypoint
env	False	list	‘’	Defined environment variables for the container
env_vars
exposed_ports	False	list	‘’	Exposes port from the container. This allows a container without an ‘EXPOSE’ directive to make it available to the host
hostname	False	string	‘’	hostname of the container
image	False	string	‘’	defined image for the container
labels	False	dict|none	{}	labels to be appended to the container
network_disabled	False	boolean	False	Disable network for the container
open_stdin	False	boolean	False	This defined multiple values: stdin_once = True attach_stdin = True detach = False
stdin_once	False	boolean	False	Opens stdin initially and closes once data transfer has been completed
tty	False	boolean	False	Open interactive pseudo tty
user	False	string	‘’	Allows the developer to set a default user to run the first process with the
volumes	False	list	None	List of volumes exposed by the container
working_dir	False	string	‘’	Starting work directory for the container
detach	False	boolean	False	Default Values applied: attach_stdout = False attach_stderr = False stdin_once = False attach_stdin = False