Maniftest File

Overview

The stackql_manifest.yml file is in the root of a project (or stack) directory. This yaml file defines all of the resources and their respective properties for all target deployment environments for your stack. Resources are processed in the order in which they are declared in this file, resources can include exports which are variables used for subsequent resources in your stack (for example a vpc_id needed to deploy a subnet). Global variables are configured here as well which can be sourced from external environment variables or secrets.

note

Secrets should not be saved in the stackql_manifest.yml file, use globals and externally sourced variables (using the -e or --env options) at deploy time.

Fields

the fields within the stackql_manifest.yml file are described in further detail here.

`name`

Type	`string`
Required	Yes

The name of the stack, by default this is derived by the init command from the stack directory name (replacing _ with - for resource and property name compliance). This name can be overridden, the value for name is exposed as a global variable called stack_name which is often used with resource or property values so ensure that this string conforms to any naming restrictions.

stackql_manifest.yml

name: kubernetes-the-hard-way

tip

Don't embed any environment symbols or designators in the name field, these are sourced at deploy time from the STACK_ENV argument to the build, test or teardown commands, and exposed for use in resource or property values as a global variable called stack_env.

`description`

Type	`string`
Required	No

Stack description, for documentation purposes only.

stackql_manifest.yml

description: stackql-deploy example for kubernetes-the-hard-way

`providers`

Type	`string[]`
Required	Yes

StackQL cloud or SaaS providers used in the stack. These are pulled from the stackql provider registry if they are not present at deploy time.

stackql_manifest.yml

providers:
  - google
  - aws

`globals`

Type	`global[]`
Required	Yes
Fields	`global.name`, `global.value`, `global.description`

Global variables used throughout the stack, can be an empty list if not required. Use the {{ YOUR_ENV_VAR }} notation in the value field of a globals list item to populate a global stack variable from an external environment variable or secret.

stackql_manifest.yml

globals:
- name: project
  description: google project name
  value: "{{ GOOGLE_PROJECT }}"
- name: region
  value: australia-southeast1
- name: default_zone
  value: australia-southeast1-a

`global.name`

Type	`string`
Required	Yes

Global variable name, this can be referred to in the resources section of the stackql_manifest.yml file or within queries using the {{ your_global_var_name }} syntax.

stackql_manifest.yml

globals:
- name: region
  value: ap-southeast-2

`global.value`

Type	`string`
Required	Yes

Global variable value, this can be a literal or sourced from an environment variable. The following example shows how to supply a string literal for the value:

stackql_manifest.yml

globals:
- name: region
  value: ap-southeast-2

The following example shows how to source value from an environment variable:

stackql_manifest.yml

globals:
- name: region
  value: {{ AWS_REGION }}

`global.description`

Type	`string`
Required	No

Global variable description, for documentation purposes only.

stackql_manifest.yml

globals:
- name: region
  description: default region  
  value: ap-southeast-2

`resources`

Type	`resource[]`
Required	Yes
Fields	`resource.name`, `resource.type`, `resource.file`, `resource.props`, `resource.exports`, `resource.protected`, `resource.description`

Defines resources in your stack, including the properties and their desired state values.

stackql_manifest.yml

resources:
- name: network
  description: vpc network for k8s-the-hard-way sample app
  props:
  - name: vpc_name
    description: name for the vpc
    value: "{{ stack_name }}-{{ stack_env }}-vpc"
  exports:
  - vpc_name    
  - vpc_link

note

A file with the name of the resource with an .iql extension is expected to exist in the resources subdirectory of your stack directory. You can reference a different file using the file field as shown here:

stackql_manifest.yml

- name: controller_instances
  file: instances.iql
  props:
  - name: num_instances
    value: 3
  - name: instance_name_prefix
    value: "{{ stack_name }}-{{ stack_env }}-controller"

`resource.name`

Type	`string`
Required	Yes

The name of the resource

stackql_manifest.yml

resources:
- name: network
...

`resource.type`

Type	`string`
Required	No

Resource type, values include : resource (default), query, script, multi

stackql_manifest.yml

resources:
- name: get_subnets
  type: query
...

info

resource will typically include create, update, delete, exists, statecheck and exports methods and is intended for provisioning or configuring a resource
query is designed to return data using an exports method
script is used to incorporate an external script in your stack definition (non StackQL query)
multi is used when resources are created in a loop (such as creating 3 vms)

`resource.file`

Type	`string`
Required	No

Query file for the resource (.iql file in the resources directory). Defaults to {resource.name}.iql

tip

Use file to reuse the same query template file for multiple different resources, as shown in the following example

stackql_manifest.yml

...
resources:
- name: internal_firewall
  file: firewalls.iql
  props:
  - name: fw_name
    value: "{{ stack_name }}-{{ stack_env }}-allow-internal-fw"
  - name: fw_direction
    value: INGRESS
  - name: fw_source_ranges
    values:
      dev:
        value: ["10.240.0.0/24", "10.200.0.0/16"]
  - name: fw_allowed
    value: [{IPProtocol: tcp}, {IPProtocol: udp}, {IPProtocol: icmp}]
- name: external_firewall
  file: firewalls.iql
  props:
  - name: fw_name
    value: "{{ stack_name }}-{{ stack_env }}-allow-external-fw"
  - name: fw_direction
    value: INGRESS
  - name: fw_source_ranges
    values:
      dev:
        value: ["0.0.0.0/0"]
  - name: fw_allowed
    value: [{IPProtocol: tcp, ports: ["22"]}, {IPProtocol: tcp, ports: ["6443"]},{IPProtocol: icmp}]
...

`resource.description`

Type	`string`
Required	No

Resource description

stackql_manifest.yml

resources:
- name: instances
  description: web server instances
...

`resource.auth`

Custom Authentication at Resource Level

This feature allows for custom authentication settings to be specified at the resource level within the stackql_manifest.yml file. This enables context-specific authentication configurations, such as control plane or data plane context switching within the same stack. Authentication parameters can be overridden by setting specific variable references in the auth section.

note

This feature requires version 1.8.0 of stackql-deploy and version 3.7.0 of pystackql, use the following to upgrade components:

stackql-deploy upgrade

Type	`object`
Required	No

The auth object will depend upon the provider the resource belongs to, consult the provider documentation in the StackQL Provider Registry Docs.

Example Usage

stackql_manifest.yml

resources:
  - name: app_manager_api_key
    props:
      - name: display_name
        value: "{{ stack_name }}-{{ stack_env }}-app-manager-api-key"
      - name: description
        value: "Kafka API Key owned by 'app-manager' service account"
      - name: owner
        value:
          id: app_manager_id
          api_version: app_manager_api_version
          kind: app_manager_kind
    exports:
      - app_manager_api_key_id
      - app_manager_api_key_secret

  - name: users_topic
    auth:
      confluent:
        type: basic
        username_var: app_manager_api_key_id
        password_var: app_manager_api_key_secret
    props:
      - name: topic_name
        value: "users"
      - name: kafka_cluster
        value: {{ cluster_id }}
      - name: rest_endpoint
        value: {{ cluster_rest_endpoint }}

This configuration sets up a custom basic authentication for the users_topic resource, where:

username_var is set to app_manager_api_key_id
password_var is set to app_manager_api_key_secret

These variables are defined in the exported section of the app_manager_api_key resource and dynamically referenced within the authentication configuration.

`resource.exports`

Type	`string[]`
Required	No

Variables exported from the resource

stackql_manifest.yml

resources:
- name: network
...
  exports:
  - vpc_name    
  - vpc_link 

note

Variables listed as exports must be returned as columns in a exports query, for example:

/*+ exports */
SELECT 
'{{ vpc_name }}' as vpc_name,
selfLink as vpc_link
FROM google.compute.networks
WHERE name = '{{ vpc_name }}'
AND project = '{{ project }}'

`resource.protected`

Type	`string[]`
Required	No

Protected variables from the resource, these variables are masked in the output logs. Protected variables are a subset of exports

stackql_manifest.yml

resources:
  - name: container_registry
...
    exports:
      - acr_url
      - acr_username
      - acr_password
    protected:
      - acr_password

`resource.props`

Type	`prop[]`
Required	Yes
Fields	`resource.prop.name`, `resource.prop.description`, `resource.prop.value`, `resource.prop.values`

Defines properties for the resource and their desired state values

stackql_manifest.yml

resources:
  - name: logging_bucket
    props:
      - name: logging_bucket_name
        value: "{{ stack_name }}-{{ stack_env }}-logging"
...

tip

You can also include environment (stack_env) selectors using values as shown here:

resources:
  - name: example_vpc
    props:
      - name: vpc_cidr_block
        values:
          prd:
            value: "10.0.0.0/16"
          sit:
            value: "10.1.0.0/16"
          dev:
            value: "10.2.0.0/16"

`resource.prop.name`

Type	`string`
Required	Yes

The name of the property

stackql_manifest.yml

- name: public_address
  props:
  - name: address_name
...

`resource.prop.description`

Type	`string`
Required	No

Property description

stackql_manifest.yml

- name: public_address
  props:
  - name: address_name
    description: web server public ip address
...

`resource.prop.value`

Type	`string`
Required	No

one of value or values must be supplied for a resource property

The value for the property

stackql_manifest.yml

- name: public_address
  props:
  - name: address_name
    value: "{{ stack_name }}-{{ stack_env }}-{{ region }}-ip-addr"  
...

`resource.prop.values`

Type	`string`
Required	No

one of value or values must be supplied for a resource property

Values for the property based upon the stack_env (stack environment selector)

stackql_manifest.yml

- name: subnetwork
  props:
  - name: ip_cidr_range
    values:
      prd:
        value: 192.168.0.0/16
      sit:
        value: 10.10.0.0/16
      dev:
        value: 10.240.0.0/24
...

`resource.prop.merge`

Type	`string[]`
Required	No

List(s) or object(s) from the context to merge with the current resource.prop.value. The merge values can be global variables defined in the globals section or exported variables from any preceding resources in the stack.

note

The resource.prop.value or values type and resource.prop.merge value(s) must be of the same type (both lists or both objects)

stackql_manifest.yml

...
globals:
  - name: region
    description: aws region
    value: "{{ AWS_REGION }}"
  - name: global_tags
    value:
      - Key: Provisioner
        Value: stackql
      - Key: StackName
        Value: "{{ stack_name }}"
      - Key: StackEnv
        Value: "{{ stack_env }}"
resources:
  - name: example_vpc
    props:
      - name: vpc_cidr_block
        values:
          prd:
            value: "10.0.0.0/16"
          sit:
            value: "10.1.0.0/16"
          dev:
            value: "10.2.0.0/16"
      - name: vpc_tags
        value:
          - Key: Name
            Value: "{{ stack_name }}-{{ stack_env }}-vpc"
        merge: 
          - global_tags
...

`version`

Type	`integer`
Required	No

Document version.

stackql_manifest.yml

version: 1

Example manifest file

Here is a complete example of a stackql_manifest.yml file for a Google stack, for other examples see the Template Library.

stackql_manifest.yml

version: 1
name: kubernetes-the-hard-way
description: stackql-deploy example for kubernetes-the-hard-way
providers:
  - google
globals:
- name: project
  description: google project name
  value: "{{ GOOGLE_PROJECT }}"
- name: region
  value: australia-southeast1
- name: default_zone
  value: australia-southeast1-a
resources:
- name: network
  description: vpc network for k8s-the-hard-way sample app
  props:
  - name: vpc_name
    description: name for the vpc
    value: "{{ stack_name }}-{{ stack_env }}-vpc"
  exports:
  - vpc_name    
  - vpc_link    
- name: subnetwork
  props:
  - name: subnet_name
    value: "{{ stack_name }}-{{ stack_env }}-{{ region }}-subnet"  
  - name: ip_cidr_range
    values:
      prd:
        value: 192.168.0.0/16
      sit:
        value: 10.10.0.0/16
      dev:
        value: 10.240.0.0/24
  exports:
    - subnet_name    
    - subnet_link            
- name: public_address
  props:
  - name: address_name
    value: "{{ stack_name }}-{{ stack_env }}-{{ region }}-ip-addr"  
  exports:
  - address    
- name: controller_instances
  file: instances.iql
  props:
  - name: num_instances
    value: 3
  - name: instance_name_prefix
    value: "{{ stack_name }}-{{ stack_env }}-controller"
  - name: disks
    value:
    - autoDelete: true
      boot: true
      initializeParams:
        diskSizeGb: 10
        sourceImage: https://compute.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/family/ubuntu-2004-lts
      mode: READ_WRITE
      type: PERSISTENT
  - name: machine_type
    value: "https://compute.googleapis.com/compute/v1/projects/{{ project }}/zones/{{ default_zone }}/machineTypes/f1-micro"          
  - name: scheduling
    value: {automaticRestart: true}
  - name: tags
    value: {items: ["{{ stack_name }}", "controller"]}
  - name: service_accounts
    value:
    - email: default
      scopes:
        - https://www.googleapis.com/auth/compute
        - https://www.googleapis.com/auth/devstorage.read_only
        - https://www.googleapis.com/auth/logging.write
        - https://www.googleapis.com/auth/monitoring
        - https://www.googleapis.com/auth/service.management.readonly
        - https://www.googleapis.com/auth/servicecontrol
  - name: network_interfaces
    values:
      dev:
        value: 
        - {networkIP: "10.240.0.10", subnetwork: "{{ subnet_link }}", accessConfigs: [{name: external-nat, type: ONE_TO_ONE_NAT}]}              
        - {networkIP: "10.240.0.11", subnetwork: "{{ subnet_link }}", accessConfigs: [{name: external-nat, type: ONE_TO_ONE_NAT}]}              
        - {networkIP: "10.240.0.12", subnetwork: "{{ subnet_link }}", accessConfigs: [{name: external-nat, type: ONE_TO_ONE_NAT}]}              
- name: worker_instances
  file: instances.iql
  props:
  - name: num_instances
    value: 3
  - name: instance_name_prefix
    value: "{{ stack_name }}-{{ stack_env }}-worker"
  - name: disks
    value:
    - autoDelete: true
      boot: true
      initializeParams:
        diskSizeGb: 10
        sourceImage: https://compute.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/family/ubuntu-2004-lts
      mode: READ_WRITE
      type: PERSISTENT
  - name: machine_type
    value: "https://compute.googleapis.com/compute/v1/projects/{{ project }}/zones/{{ default_zone }}/machineTypes/f1-micro"          
  - name: scheduling
    value: {automaticRestart: true}
  - name: tags
    value: {items: ["{{ stack_name }}", "worker"]}
  - name: service_accounts
    value:
    - email: default
      scopes:
        - https://www.googleapis.com/auth/compute
        - https://www.googleapis.com/auth/devstorage.read_only
        - https://www.googleapis.com/auth/logging.write
        - https://www.googleapis.com/auth/monitoring
        - https://www.googleapis.com/auth/service.management.readonly
        - https://www.googleapis.com/auth/servicecontrol
  - name: network_interfaces
    values:
      dev:
        value: 
        - {networkIP: "10.240.0.20", subnetwork: "{{ subnet_link }}", accessConfigs: [{name: external-nat, type: ONE_TO_ONE_NAT}]}              
        - {networkIP: "10.240.0.21", subnetwork: "{{ subnet_link }}", accessConfigs: [{name: external-nat, type: ONE_TO_ONE_NAT}]}              
        - {networkIP: "10.240.0.22", subnetwork: "{{ subnet_link }}", accessConfigs: [{name: external-nat, type: ONE_TO_ONE_NAT}]} 
- name: health_checks
  props:
  - name: health_check_name
    value: kubernetes
  - name: health_check_interval_sec
    value: 5
  - name: health_check_description
    value: Kubernetes Health Check
  - name: health_check_timeout_sec
    value: 5
  - name: health_check_healthy_threshold
    value: 2
  - name: health_check_unhealthy_threshold
    value: 2
  - name: health_check_host
    value: kubernetes.default.svc.cluster.local
  - name: health_check_port
    value: 80
  - name: health_check_path
    value: /healthz
  exports:
    - health_check_link
- name: internal_firewall
  file: firewalls.iql
  props:
  - name: fw_name
    value: "{{ stack_name }}-{{ stack_env }}-allow-internal-fw"
  - name: fw_direction
    value: INGRESS
  - name: fw_source_ranges
    values:
      dev:
        value: ["10.240.0.0/24", "10.200.0.0/16"]
  - name: fw_allowed
    value: [{IPProtocol: tcp}, {IPProtocol: udp}, {IPProtocol: icmp}]
- name: external_firewall
  file: firewalls.iql
  props:
  - name: fw_name
    value: "{{ stack_name }}-{{ stack_env }}-allow-external-fw"
  - name: fw_direction
    value: INGRESS
  - name: fw_source_ranges
    values:
      dev:
        value: ["0.0.0.0/0"]
  - name: fw_allowed
    value: [{IPProtocol: tcp, ports: ["22"]}, {IPProtocol: tcp, ports: ["6443"]},{IPProtocol: icmp}]
- name: health_check_firewall
  file: firewalls.iql
  props:
  - name: fw_name
    value: "{{ stack_name }}-{{ stack_env }}-allow-health-check-fw"
  - name: fw_direction
    value: INGRESS
  - name: fw_source_ranges
    values:
      dev:
        value: ["209.85.152.0/22", "209.85.204.0/22", "35.191.0.0/16"]
  - name: fw_allowed
    value: [{IPProtocol: tcp}]
- name: get_controller_instances
  type: query
  exports:
    - controller_instances
- name: target_pool
  props:
  - name: target_pool_name
    value: "{{ stack_name }}-{{ stack_env }}-target-pool"
  - name: target_pool_session_affinity
    value: NONE
  - name: target_pool_health_checks
    value: ["{{ health_check_link }}"]
  - name: target_pool_instances
    value: "{{ controller_instances }}"
  exports:
    - target_pool_link
- name: forwarding_rule
  props:
  - name: forwarding_rule_name
    value: "{{ stack_name }}-{{ stack_env }}-forwarding-rule"
  - name: forwarding_rule_load_balancing_scheme
    value: EXTERNAL
  - name: forwarding_rule_port_range
    value: 6443
- name: routes
  props:
  - name: num_routes
    value: 3
  - name: route_name_prefix
    value: "{{ stack_name }}-{{ stack_env }}-route"
  - name: route_priority
    value: 1000
  - name: route_data
    values:
      dev:
        value: 
        - {dest_range: "10.200.0.0/24", next_hop_ip: "10.240.0.20"}              
        - {dest_range: "10.200.1.0/24", next_hop_ip: "10.240.0.21"}              
        - {dest_range: "10.200.2.0/24", next_hop_ip: "10.240.0.22"}              

Overview​

Fields​

name​

description​

providers​

globals​

global.name​

global.value​

global.description​

resources​

resource.name​

resource.type​

resource.file​

resource.description​

resource.auth​

Custom Authentication at Resource Level​

Example Usage​

resource.exports​

resource.protected​

resource.props​

resource.prop.name​

resource.prop.description​

resource.prop.value​

resource.prop.values​

resource.prop.merge​

version​

Example manifest file​

Overview

Fields

`name`

`description`

`providers`

`globals`

`global.name`

`global.value`

`global.description`

`resources`

`resource.name`

`resource.type`

`resource.file`

`resource.description`

`resource.auth`

Custom Authentication at Resource Level

Example Usage

`resource.exports`

`resource.protected`

`resource.props`

`resource.prop.name`

`resource.prop.description`

`resource.prop.value`

`resource.prop.values`

`resource.prop.merge`

`version`

Example manifest file