Cloud Composer #Deploy with CloudShell
This sandbox will deploy a Cloud Composer
List of the components used in this sandbox:
Note: additional details are available in Cloud Composer Guide
NOTE: If you want to undeploy or update an existing sandbox please proceed directly to Step 3 - Recover
First you need initialize the sandbox configuration. To do this please run the initialization command:
hub stack init
The command will prompt you to enter the
GCP project ID and
will create initial configuration files and download required components.
Next, to configure sandbox manually run the following command:
hub stack configure
The Hub CLI produces a state file after provisioning of sandbox,
same as its components, and it must be uploaded to the cloud.
For simplicity, we pre-create
Cloud DNS Zone and
in your GCP project with pre-generated names.
This is done during
Deploy Sandbox #
Once you are done with the configuration, use the following command to deploy the sandbox:
hub stack deploy
If you want to change environemnt configuration and redeploy a composer environment run following command:
hub stack undeploy -c "composer"
Make a change in the sandbox configuration or in the body of the component and run next command:
hub stack deploy -c "composer"
Note: even in minimal configuration it will still take 20+ minutes to redeploy
component has a set of parameters (key-value pairs) such as
GKE cluster name or number of nodes in a node pool, etc.
parameters section of the
hub.yaml to see what parameters are available.
Any default value can be overridden.
To get more information on how components are implemented refer to their README files.
To view parameters for the current sandbox run this command:
|Number of GKE nodes for compose environment. Must be at least |
|Number of GKE nodes for compose environment|
|Version of composer (|
|Target environment subnet address range|
You can read about configuration options here: https://cloud.google.com/composer/docs/how-to/managing/creating
Undeploy Sandbox #
To delete the sandbox, run the followng command:
hub stack undeploy
Google Cloud Shell environments are ephemeral, which means any files stored locally on the Cloud Shell machine will be lost when the machine restarts. Luckily, we store sandbox state files in the Cloud, which means, if we lose a local state, we can always recover it from the Cloud.
To recover your sandbox please follow the steps below.
Switch to your current project:
gcloud config set project <project-id>
List all recent sandboxes deployed to your project and find ID of the sandbox you would like to recover:
hub state ls
Initialize your sandbox:
hub stack init <id>
Hub CLI will find the location of the state file, download it and prepare configuration files.
After stack recovery undeploy or update your existing stack using corresponding commands.
Getting deployment result #
After successful deployment you will get three URLs
- Airflow URL
- GCS bucket
- GKE Cluster
You can also find it by running command:
If you want to see only the configuration parameters used by Composer run the following command
hub show -c "composer"
Get GKE kubeconfig #
Use a query string for
cluster_name="$(hub show -q '.parameters.composer.gkeCluster' -- -Mr)" gcloud container clusters get-credentials "$cluster_name" # to confirm cluster setup k cluster-info
TODO: we may want to delegate this to the
post-deploy. Depends if we bother to deploy on top anything else, (i.e. Ingress Controller, user apps etc as set of components)