Maniftest File
Overview
The stackql_manifest.yml file is in the root of a project (or stack) directory. This yaml file defines all of the resources and their respective properties for all target deployment environments for your stack. Resources are processed in the order in which they are declared in this file, resources can include exports which are variables used for subsequent resources in your stack (for example a vpc_id needed to deploy a subnet). Global variables are configured here as well which can be sourced from external environment variables or secrets.
Secrets should not be saved in the stackql_manifest.yml file, use globals and externally sourced variables (using the -e or --env options) at deploy time.
Fields
the fields within the stackql_manifest.yml file are described in further detail here.
name
| Type | string |
| Required | Yes |
The name of the stack, by default this is derived by the init command from the stack directory name (replacing _ with - for resource and property name compliance). This name can be overridden, the value for name is exposed as a global variable called stack_name which is often used with resource or property values so ensure that this string conforms to any naming restrictions.
name: kubernetes-the-hard-way
Don't embed any environment symbols or designators in the name field, these are sourced at deploy time from the STACK_ENV argument to the build, test or teardown commands, and exposed for use in resource or property values as a global variable called stack_env.
description
| Type | string |
| Required | No |
Stack description, for documentation purposes only.
description: stackql-deploy example for kubernetes-the-hard-way
providers
| Type | string[] |
| Required | Yes |
StackQL cloud or SaaS providers used in the stack. These are pulled from the stackql provider registry if they are not present at deploy time.
providers:
- google
- aws
globals
| Type | global[] |
| Required | Yes |
| Fields | global.name, global.value, global.description |
Global variables used throughout the stack, can be an empty list if not required. Use the {{ YOUR_ENV_VAR }} notation in the value field of a globals list item to populate a global stack variable from an external environment variable or secret.
globals:
- name: project
description: google project name
value: "{{ GOOGLE_PROJECT }}"
- name: region
value: australia-southeast1
- name: default_zone
value: australia-southeast1-a
global.name
| Type | string |
| Required | Yes |
Global variable name, this can be referred to in the resources section of the stackql_manifest.yml file or within queries using the {{ your_global_var_name }} syntax.
globals:
- name: region
value: ap-southeast-2
global.value
| Type | string |
| Required | Yes |
Global variable value, this can be a literal or sourced from an environment variable. The following example shows how to supply a string literal for the value:
globals:
- name: region
value: ap-southeast-2
The following example shows how to source value from an environment variable:
globals:
- name: region
value: {{ AWS_REGION }}
global.description
| Type | string |
| Required | No |
Global variable description, for documentation purposes only.
globals:
- name: region
description: default region
value: ap-southeast-2
resources
| Type | resource[] |
| Required | Yes |
| Fields | resource.name, resource.type, resource.file, resource.props, resource.exports, resource.protected, resource.description, resource.if, resource.auth, resource.sql, resource.skipvalidation |
Defines resources in your stack, including the properties and their desired state values.
resources:
- name: network
description: vpc network for k8s-the-hard-way sample app
props:
- name: vpc_name
description: name for the vpc
value: "{{ stack_name }}-{{ stack_env }}-vpc"
exports:
- vpc_name
- vpc_link
A file with the name of the resource with an .iql extension is expected to exist in the resources subdirectory of your stack directory. You can reference a different file using the file field as shown here:
- name: controller_instances
file: instances.iql
props:
- name: num_instances
value: 3
- name: instance_name_prefix
value: "{{ stack_name }}-{{ stack_env }}-controller"
resource.name
| Type | string |
| Required | Yes |
The name of the resource
resources:
- name: network
...
resource.type
| Type | string |
| Required | No |
Resource type, values include : resource (default), query, script, multi, command
resources:
- name: get_subnets
type: query
...
resourcewill typically includecreate,update,delete,exists,statecheckandexportsmethods and is intended for provisioning or configuring a resourcequeryis designed to return data using anexportsmethodscriptis used to incorporate an external script in your stack definition (non StackQL query)multiis used when resources are created in a loop (such as creating 3 vms)commandis used to run a command (like anUPDATE,INSERT) during abuildoperation (which does not export any variables)
resource.file
| Type | string |
| Required | No |
Query file for the resource (.iql file in the resources directory). Defaults to {resource.name}.iql
Use file to reuse the same query template file for multiple different resources, as shown in the following example
...
resources:
- name: internal_firewall
file: firewalls.iql
props:
- name: fw_name
value: "{{ stack_name }}-{{ stack_env }}-allow-internal-fw"
- name: fw_direction
value: INGRESS
- name: fw_source_ranges
values:
dev:
value: ["10.240.0.0/24", "10.200.0.0/16"]
- name: fw_allowed
value: [{IPProtocol: tcp}, {IPProtocol: udp}, {IPProtocol: icmp}]
- name: external_firewall
file: firewalls.iql
props:
- name: fw_name
value: "{{ stack_name }}-{{ stack_env }}-allow-external-fw"
- name: fw_direction
value: INGRESS
- name: fw_source_ranges
values:
dev:
value: ["0.0.0.0/0"]
- name: fw_allowed
value: [{IPProtocol: tcp, ports: ["22"]}, {IPProtocol: tcp, ports: ["6443"]},{IPProtocol: icmp}]
...
resource.description
| Type | string |
| Required | No |
Resource description
resources:
- name: instances
description: web server instances
...
resource.auth
Custom Authentication at Resource Level
This feature allows for custom authentication settings to be specified at the resource level within the stackql_manifest.yml file. This enables context-specific authentication configurations, such as control plane or data plane context switching within the same stack. Authentication parameters can be overridden by setting specific variable references in the auth section.
This feature requires version 1.8.0 of stackql-deploy and version 3.7.0 of pystackql, use the following to upgrade components:
stackql-deploy upgrade
| Type | object |
| Required | No |
The auth object will depend upon the provider the resource belongs to, consult the provider documentation in the StackQL Provider Registry Docs.
Example Usage
resources:
- name: app_manager_api_key
props:
- name: display_name
value: "{{ stack_name }}-{{ stack_env }}-app-manager-api-key"
- name: description
value: "Kafka API Key owned by 'app-manager' service account"
- name: owner
value:
id: app_manager_id
api_version: app_manager_api_version
kind: app_manager_kind
exports:
- app_manager_api_key_id
- app_manager_api_key_secret
- name: users_topic
auth:
confluent:
type: basic
username_var: app_manager_api_key_id
password_var: app_manager_api_key_secret
props:
- name: topic_name
value: "users"
- name: kafka_cluster
value: {{ cluster_id }}
- name: rest_endpoint
value: {{ cluster_rest_endpoint }}
This configuration sets up a custom basic authentication for the users_topic resource, where:
username_varis set toapp_manager_api_key_idpassword_varis set toapp_manager_api_key_secret
These variables are defined in the exported section of the app_manager_api_key resource and dynamically referenced within the authentication configuration.
resource.exports
| Type | string[] |
| Required | No |
Variables exported from the resource
resources:
- name: network
...
exports:
- vpc_name
- vpc_link
Variables listed as exports must be returned as columns in a exports query, for example:
/*+ exports */
SELECT
'{{ vpc_name }}' as vpc_name,
selfLink as vpc_link
FROM google.compute.networks
WHERE name = '{{ vpc_name }}'
AND project = '{{ project }}'
resource.protected
| Type | string[] |
| Required | No |
Protected variables from the resource, these variables are masked in the output logs. Protected variables are a subset of exports
resources:
- name: container_registry
...
exports:
- acr_url
- acr_username
- acr_password
protected:
- acr_password
resource.if
| Type | string |
| Required | No |
A conditional expression that determines whether a resource should be tested, provisioned, or deprovisioned. You can use Python expressions to conditionally determine if a resource should be processed.
resources:
- name: get_transfer_kms_key_id
if: "environment == 'production'"
...
- Conditions are evaluated as Python expressions.
- You can reference literals (string, boolean, integer, etc.) or runtime template variables.
- If the condition evaluates to
True, the resource is processed; ifFalse, it is skipped. - Template variables can be referenced using Jinja2 template syntax (
{{ variable }}).
Examples
Conditionally process a resource based on environment:
resources:
- name: get_transfer_kms_key_id
if: "environment == 'production'"
...
Conditionally process based on other variable values:
resources:
- name: get_transfer_kms_key_id
if: "some_var == '{{ some_other_var_value }}'"
...
resource.sql
| Type | string |
| Required | No |
For command and query type resources, you can include SQL statements directly in your resource manifest using the sql key. This allows you to write custom SQL commands without needing a separate IQL file.
resources:
- name: databricks_workspace/unitycatalog/grants
type: command
props: [...]
sql: |
/*+ update */
UPDATE databricks_workspace.unitycatalog.grants
SET principal = '{{ principal }}',
privileges = {{ privileges }}
WHERE full_name = '{{ full_name }}' AND
securable_type = '{{ securable_type }}' AND
deployment_name = '{{ deployment_name }}';
- The
sqlkey is only supported tocommandandquerytype resources - For command resources, either
sqlor a corresponding IQL file with acommandanchor must be provided, ifsqlis supplied in the manifest this will be used - The
sqlkey accepts a string containing the SQL statement to execute - You can use multi-line strings with the YAML pipe (
|) character for better readability - Template variables can be referenced using Jinja2 template syntax (
{{ variable }})
When to Use
The sql key is particularly useful for:
- Simple commands that don't warrant a separate IQL file
- One-off operations specific to a particular deployment
- Custom operations like granting permissions in Unity Catalog
Examples
Grant Permissions in Unity Catalog
- name: databricks_workspace/unitycatalog/grants
type: command
props:
- name: full_name
value: "my-storage-credential"
- name: securable_type
value: "storage_credential"
- name: deployment_name
value: "{{ databricks_deployment_name }}"
- name: principal
value: "account users"
- name: privileges
value:
- "CREATE_EXTERNAL_LOCATION"
- "USE"
sql: |
/*+ update */
UPDATE databricks_workspace.unitycatalog.grants
SET principal = '{{ principal }}',
privileges = {{ privileges }}
WHERE full_name = '{{ full_name }}' AND
securable_type = '{{ securable_type }}' AND
deployment_name = '{{ deployment_name }}';
Run a Custom Query with Conditional Logic
You can combine the sql key with conditional processing:
- name: custom_command
type: command
if: "environment == 'production'"
props:
- name: table_name
value: "{{ stack_name }}_audit_log"
sql: |
/*+ update */
INSERT INTO {{ table_name }}
VALUES ('{{ stack_name }}', '{{ stack_env }}', '{{ deployment_timestamp }}');
resource.skip_validation
| Type | boolean |
| Required | No |
When set to true, the test and build commands will bypass validation checks for this specific resource. This is particularly useful for resources that are initially created with placeholder values and later updated within the same stack.
resources:
- name: aws/iam/metastore_access_role
file: aws/iam/iam_role.iql
props:
- name: role_name
value: "{{ stack_name }}-{{ stack_env }}-metastore-role"
- name: assume_role_policy_document
value:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
AWS:
- "arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL"
Action: "sts:AssumeRole"
Condition:
StringEquals:
sts:ExternalId: "0000" # Placeholder
skip_validation: true
exports:
- aws_iam_role_arn: metastore_access_role_arn
- Use
skip_validation: truewhen you need to create a resource with temporary configuration that will be updated later in the stack execution - Common use cases include:
- Creating IAM roles with placeholder external IDs that will be updated once another dependent resource is created
- Setting up initial placeholder credentials that will be modified in a subsequent step
- Creating resources with circular dependencies where initial validation would fail
- This flag only affects the
testandbuildcommands
resource.props
| Type | prop[] |
| Required | Yes |
| Fields | resource.prop.name, resource.prop.description, resource.prop.value, resource.prop.values |
Defines properties for the resource and their desired state values
resources:
- name: logging_bucket
props:
- name: logging_bucket_name
value: "{{ stack_name }}-{{ stack_env }}-logging"
...
You can also include environment (stack_env) selectors using values as shown here:
resources:
- name: example_vpc
props:
- name: vpc_cidr_block
values:
prd:
value: "10.0.0.0/16"
sit:
value: "10.1.0.0/16"
dev:
value: "10.2.0.0/16"
resource.prop.name
| Type | string |
| Required | Yes |
The name of the property
- name: public_address
props:
- name: address_name
...
resource.prop.description
| Type | string |
| Required | No |
Property description
- name: public_address
props:
- name: address_name
description: web server public ip address
...
resource.prop.value
| Type | string |
| Required | No |
one of
valueorvaluesmust be supplied for a resource property
The value for the property
- name: public_address
props:
- name: address_name
value: "{{ stack_name }}-{{ stack_env }}-{{ region }}-ip-addr"
...
resource.prop.values
| Type | string |
| Required | No |
one of
valueorvaluesmust be supplied for a resource property
Values for the property based upon the stack_env (stack environment selector)
- name: subnetwork
props:
- name: ip_cidr_range
values:
prd:
value: 192.168.0.0/16
sit:
value: 10.10.0.0/16
dev:
value: 10.240.0.0/24
...
resource.prop.merge
| Type | string[] |
| Required | No |
List(s) or object(s) from the context to merge with the current resource.prop.value. The merge values can be global variables defined in the globals section or exported variables from any preceding resources in the stack.
The resource.prop.value or values type and resource.prop.merge value(s) must be of the same type (both lists or both objects)
...
globals:
- name: region
description: aws region
value: "{{ AWS_REGION }}"
- name: global_tags
value:
- Key: Provisioner
Value: stackql
- Key: StackName
Value: "{{ stack_name }}"
- Key: StackEnv
Value: "{{ stack_env }}"
resources:
- name: example_vpc
props:
- name: vpc_cidr_block
values:
prd:
value: "10.0.0.0/16"
sit:
value: "10.1.0.0/16"
dev:
value: "10.2.0.0/16"
- name: vpc_tags
value:
- Key: Name
Value: "{{ stack_name }}-{{ stack_env }}-vpc"
merge:
- global_tags
...
exports
Type: array of strings (optional)
Description: List of variable names to export to a JSON file after deployment completion. Variables must exist in the deployment context (from globals or resource exports). Use with the --output-file CLI argument to specify the destination file.
Usage: Use this to extract key deployment outputs for use in CI/CD pipelines, downstream processes, or for record-keeping.
Example:
exports:
- databricks_workspace_name
- databricks_workspace_id
- aws_iam_role_arn
- deployment_timestamp
Notes:
stack_name,stack_env, andelapsed_timeare automatically included in exports and do not need to be listed- Export order: automatic exports (
stack_name,stack_env) first, then user-defined exports, then timing (elapsed_time) last elapsed_timeis formatted as a string showing the total deployment duration (e.g., "0:01:23.456789")- Variables are exported exactly as they exist in the deployment context
- Complex objects and arrays are preserved as JSON structures
- If a listed variable doesn't exist in the context, deployment will fail
- Requires
--output-fileCLI argument to be specified, otherwise exports are skipped - Exported JSON file contains a flat object with variable names as keys
version
| Type | integer |
| Required | No |
Document version.
version: 1
Example manifest file
Here is a complete example of a stackql_manifest.yml file for a Google stack, for other examples see the Template Library.
version: 1
name: kubernetes-the-hard-way
description: stackql-deploy example for kubernetes-the-hard-way
providers:
- google
globals:
- name: project
description: google project name
value: "{{ GOOGLE_PROJECT }}"
- name: region
value: australia-southeast1
- name: default_zone
value: australia-southeast1-a
resources:
- name: network
description: vpc network for k8s-the-hard-way sample app
props:
- name: vpc_name
description: name for the vpc
value: "{{ stack_name }}-{{ stack_env }}-vpc"
exports:
- vpc_name
- vpc_link
- name: subnetwork
props:
- name: subnet_name
value: "{{ stack_name }}-{{ stack_env }}-{{ region }}-subnet"
- name: ip_cidr_range
values:
prd:
value: 192.168.0.0/16
sit:
value: 10.10.0.0/16
dev:
value: 10.240.0.0/24
exports:
- subnet_name
- subnet_link
- name: public_address
props:
- name: address_name
value: "{{ stack_name }}-{{ stack_env }}-{{ region }}-ip-addr"
exports:
- address
- name: controller_instances
file: instances.iql
props:
- name: num_instances
value: 3
- name: instance_name_prefix
value: "{{ stack_name }}-{{ stack_env }}-controller"
- name: disks
value:
- autoDelete: true
boot: true
initializeParams:
diskSizeGb: 10
sourceImage: https://compute.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/family/ubuntu-2004-lts
mode: READ_WRITE
type: PERSISTENT
- name: machine_type
value: "https://compute.googleapis.com/compute/v1/projects/{{ project }}/zones/{{ default_zone }}/machineTypes/f1-micro"
- name: scheduling
value: {automaticRestart: true}
- name: tags
value: {items: ["{{ stack_name }}", "controller"]}
- name: service_accounts
value:
- email: default
scopes:
- https://www.googleapis.com/auth/compute
- https://www.googleapis.com/auth/devstorage.read_only
- https://www.googleapis.com/auth/logging.write
- https://www.googleapis.com/auth/monitoring
- https://www.googleapis.com/auth/service.management.readonly
- https://www.googleapis.com/auth/servicecontrol
- name: network_interfaces
values:
dev:
value:
- {networkIP: "10.240.0.10", subnetwork: "{{ subnet_link }}", accessConfigs: [{name: external-nat, type: ONE_TO_ONE_NAT}]}
- {networkIP: "10.240.0.11", subnetwork: "{{ subnet_link }}", accessConfigs: [{name: external-nat, type: ONE_TO_ONE_NAT}]}
- {networkIP: "10.240.0.12", subnetwork: "{{ subnet_link }}", accessConfigs: [{name: external-nat, type: ONE_TO_ONE_NAT}]}
- name: worker_instances
file: instances.iql
props:
- name: num_instances
value: 3
- name: instance_name_prefix
value: "{{ stack_name }}-{{ stack_env }}-worker"
- name: disks
value:
- autoDelete: true
boot: true
initializeParams:
diskSizeGb: 10
sourceImage: https://compute.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/family/ubuntu-2004-lts
mode: READ_WRITE
type: PERSISTENT
- name: machine_type
value: "https://compute.googleapis.com/compute/v1/projects/{{ project }}/zones/{{ default_zone }}/machineTypes/f1-micro"
- name: scheduling
value: {automaticRestart: true}
- name: tags
value: {items: ["{{ stack_name }}", "worker"]}
- name: service_accounts
value:
- email: default
scopes:
- https://www.googleapis.com/auth/compute
- https://www.googleapis.com/auth/devstorage.read_only
- https://www.googleapis.com/auth/logging.write
- https://www.googleapis.com/auth/monitoring
- https://www.googleapis.com/auth/service.management.readonly
- https://www.googleapis.com/auth/servicecontrol
- name: network_interfaces
values:
dev:
value:
- {networkIP: "10.240.0.20", subnetwork: "{{ subnet_link }}", accessConfigs: [{name: external-nat, type: ONE_TO_ONE_NAT}]}
- {networkIP: "10.240.0.21", subnetwork: "{{ subnet_link }}", accessConfigs: [{name: external-nat, type: ONE_TO_ONE_NAT}]}
- {networkIP: "10.240.0.22", subnetwork: "{{ subnet_link }}", accessConfigs: [{name: external-nat, type: ONE_TO_ONE_NAT}]}
- name: health_checks
props:
- name: health_check_name
value: kubernetes
- name: health_check_interval_sec
value: 5
- name: health_check_description
value: Kubernetes Health Check
- name: health_check_timeout_sec
value: 5
- name: health_check_healthy_threshold
value: 2
- name: health_check_unhealthy_threshold
value: 2
- name: health_check_host
value: kubernetes.default.svc.cluster.local
- name: health_check_port
value: 80
- name: health_check_path
value: /healthz
exports:
- health_check_link
- name: internal_firewall
file: firewalls.iql
props:
- name: fw_name
value: "{{ stack_name }}-{{ stack_env }}-allow-internal-fw"
- name: fw_direction
value: INGRESS
- name: fw_source_ranges
values:
dev:
value: ["10.240.0.0/24", "10.200.0.0/16"]
- name: fw_allowed
value: [{IPProtocol: tcp}, {IPProtocol: udp}, {IPProtocol: icmp}]
- name: external_firewall
file: firewalls.iql
props:
- name: fw_name
value: "{{ stack_name }}-{{ stack_env }}-allow-external-fw"
- name: fw_direction
value: INGRESS
- name: fw_source_ranges
values:
dev:
value: ["0.0.0.0/0"]
- name: fw_allowed
value: [{IPProtocol: tcp, ports: ["22"]}, {IPProtocol: tcp, ports: ["6443"]},{IPProtocol: icmp}]
- name: health_check_firewall
file: firewalls.iql
props:
- name: fw_name
value: "{{ stack_name }}-{{ stack_env }}-allow-health-check-fw"
- name: fw_direction
value: INGRESS
- name: fw_source_ranges
values:
dev:
value: ["209.85.152.0/22", "209.85.204.0/22", "35.191.0.0/16"]
- name: fw_allowed
value: [{IPProtocol: tcp}]
- name: get_controller_instances
type: query
exports:
- controller_instances
- name: target_pool
props:
- name: target_pool_name
value: "{{ stack_name }}-{{ stack_env }}-target-pool"
- name: target_pool_session_affinity
value: NONE
- name: target_pool_health_checks
value: ["{{ health_check_link }}"]
- name: target_pool_instances
value: "{{ controller_instances }}"
exports:
- target_pool_link
- name: forwarding_rule
props:
- name: forwarding_rule_name
value: "{{ stack_name }}-{{ stack_env }}-forwarding-rule"
- name: forwarding_rule_load_balancing_scheme
value: EXTERNAL
- name: forwarding_rule_port_range
value: 6443
- name: routes
props:
- name: num_routes
value: 3
- name: route_name_prefix
value: "{{ stack_name }}-{{ stack_env }}-route"
- name: route_priority
value: 1000
- name: route_data
values:
dev:
value:
- {dest_range: "10.200.0.0/24", next_hop_ip: "10.240.0.20"}
- {dest_range: "10.200.1.0/24", next_hop_ip: "10.240.0.21"}
- {dest_range: "10.200.2.0/24", next_hop_ip: "10.240.0.22"}
exports:
- vpc_name
- vpc_link
- subnet_name
- address
- target_pool_link