Cloud Functions and Cloud Tasks, the ultimate guide

Reddit
Linkedin

Pre-order use case

The repo is here.

One of the missing pieces of the google cloud serverless stack was the ability to automate future tasks dynamically (not the cron jobs), in the pass, I used to set a Redis instance and play around with the notify-keyspace-events feature, so if I needed for example, to clean some cached data per user, or exec specific actions at a dynamic given time, it was not possible; I event tried to find a way to triger PubSub events from MemoryStore., not possible. I tried to find any documentation about integrating with Cloud Functions and I couldn't find any, so here I am.

This is where Google Cloud Tasks shine. Cloud Tasks can perform a large number of distributed tasks like a distributed queues, allow you to perform work asynchronously. A few use cases can be:

  • Send notifications to an user about an about to expire payment.
  • Process pre-order payments after a product release.
  • Trigger others Services at a given time (BidQuery, AppEngine, Cloud Functions, Hit any endpoint, etc).
  • Send welcome emails 24 hours (or any) after a user is sign up.

The most useful case which make the difference vs Schedule Functions, it allows to execute 1 or more times functions per specific users at any given time.

I assume that if you are here, you already know what you are looking for, so let's make the flow. We have to:

  • Create a Cloud Tasks Queue.
  • Create a Cloud Function which will create a Cloud Tasks on Demand. (We need a trigger for this)
  • Create a HTTP Cloud Function with the business logic.
  • Ensure this Cloud Tasks trigger HTTP Cloud Function.
  • Ensure the HTTP Cloud Function is only accessible through the Cloud Tasks (and not the outside world).

diagram

Create a Cloud Tasks Queue.

Make sure you enable the Cloud Tasks API via API & Services -> Drashboard -> ENABLE APIS AND SERVICES and search for Cloud Tasks API. By the time I'm writing this post, I couldn't find a way to to manage queue creation via the Console, so you need to get your hands into the Google Cloud SDK.

Let's create our queue, it will be called sample-queue 🤷🏻‍♂️, cause, why not?

gcloud tasks queues create sample-queue

Done! You can check more about this queue typing gcloud tasks queues describe sample-queue, and if you go to the Console, it will look like this.

console

Create a Cloud Function which will create Cloud Tasks on Demand

There are a few types of tasks, the one we are interested for this example are HTTP tasks. we are going to take advantage of the Nodejs SDK to create tasks and add it to the queue on the demand, the code looks like this:

// functions/src/tasks/create.ts
'use strict'

const project = process.env.GCP_PROJECT
const location = `us-east1` // ! hardcode Google Task location, this is NOT the function location
const function_url = `https://us-central1-cloud-tasks-sample-263214.cloudfunctions.net/preOrder` // ! change this for your task executer function
const default_queue = `sample-queue` // ! change this for your queue name
const SERVICE_ACCOUNT_EMAIL = `http-function-invoker-only@cloud-tasks-sample-263214.iam.gserviceaccount.com` // ! change this for your service account email
const default_task = `pre_order`

export const createTasks = async function (
    payload: any, 
    date: string, 
    task_name: string = default_task, 
    queue_name: string = default_queue, 
    default_function: string = function_url
  ) {
  const { CloudTasksClient } = require('@google-cloud/tasks')
  const client = new CloudTasksClient()
  const parent = client.queuePath(project, location, queue_name)

  const convertedPayload = JSON.stringify(payload)
  const body = Buffer.from(convertedPayload).toString('base64')

  const taskName = `${task_name}_${payload.id}` // ! must be unique

  const task: any = {
    httpRequest: {
      httpMethod: 'POST',
      url: default_function,
      headers: {
        'Content-Type': 'application/json',
      },
      body,
      oidcToken: {
        serviceAccountEmail: SERVICE_ACCOUNT_EMAIL
      }
    },
    name: `projects/${project}/locations/${location}/queues/${queue_name}/tasks/${taskName}`
  }

  // ! dates set in the pass will be set as current date
  const convertedDate = new Date(date)
  const currentDate = new Date()

  if (convertedDate < currentDate)
    throw new Error(`Scheduled data in the past.`)

  const date_in_release_in_seconds = convertedDate.getTime() / 1000

  task.scheduleTime = {
    seconds: date_in_release_in_seconds,
  };

  const [ response ] = await client.createTask({ parent, task })

  console.log(`Created task ${response.name}`)

  return Promise.resolve({ task: response.name })
}

Let's elaborate a little, there are 2 arguments we need to create a task:

  • Payload (what information we need to pass).
  • Date (When this tasks is going to be executed).

Then we have Queue information arguments:

  • Task Name (this need to be unique per each tasks, otherwise, it will throw an Already Exists Error).
  • Queue Name (which queue are you going to use).
  • Function endpoint (the http location this task is going to trigger). We will create this on the next step.

Be aware, you can trigger any endpoint, not necessary a HTTP Cloud Function, it will contain the payload on the body request.

You also will need a Service Account, we will use it to restrict the HTTP Cloud Function to be trigger only by this task, more about this in a minute, hand in there.

Also, the Location refers to the Cloud Tasks Location, not the Cloud Function Location.

Now we need to create a function which will create these tasks. Let's keep it simple for the sake of the example:

export const createPreOrder = functions.firestore.document(`order/{doc}`).onCreate(async (snap) => {
  const data = snap.data()
  const id = snap.id
  const date = data.date // "2020-01-25T15:00:00.000Z"
  const { task } = await createTasks({
    ...data,
    id, // we use this to guarantee the uniqueness of the tasks
  }, date)

  console.log(task)

  return Promise.resolve({ ok: true })
})

We are going to assume that each time a document is created on an order collection, it will contain a date property and all the information we need to pass to the Task. Very simple, we can use this sample for example To Process a payment after a product is released, a Pre Order feature.

Assuming you create a few tasks, you console should look something like this:

console-two

Create a HTTP Cloud Function with the business logic.

This is the function we are going to execute, let's see the generic code, the business logic is up to your use case.

export const preOrder = functions.https.onRequest((request, response) => {
 console.log(request.body) 

 /**
  * 
  * 
  * Do whatever you need with the payload HERE
  * 
  * 
  * 
  **/
  
 try {
    // ! Send OK to Cloud Task queue to delete task.
    response.status(200).send('Task Completed');
  } catch (error) {
    // ! Any status code other than 2xx or 503 will trigger the task to retry.
    response.status(error.code).send(error.message);
  }
})

Remember to copy the url of this function and update your create tasks with this information.

Task will pass the payload, so it will be available via request.body, you can implement whatever logic you need from here. You need to return a status code 200 to remove the Task from the queue, otherwise, it will go into a Retry mode (You can configure this behavior via the SDK).

SWEET! now you can make some tests using the Run Now button on the tasks via the Cloud Tasks Console.

Secure the HTTP Functions

You are probably wonder, yo but that HTTP function is open to the outside worldundefined wtf? I know you are smart and that SOC2 audit won't pass by itself right?

Sadly (at this moment), the Firebase SDK does not support per functions IAM roles, so we need to get our hands dirty on the CLI again, let's do it!

Create A Service Account

Remember a few steps ago I mentioned this? on the task payload, we have something which looks like this:

const SERVICE_ACCOUNT_EMAIL = `?`
const task = {
  //...
  oidcToken: {
    serviceAccountEmail: SERVICE_ACCOUNT_EMAIL
  }
}

You can use the Google sdk to create a Service Account, but let's use the console for simplicity, let's go to IAM & Admin -> Services accounts

service-account

Let's create a service Account, it should look something like this:

service-account-two

You can apply the Cloud Functions Invoker role but let's do it better via the CLI instead so we can ensure it will only apply only to the HTTP function, so let the role empty.

Now your keys should look something like this:

service-account-three

Copy this email the email and update the SERVICE_ACCOUNT_EMAIL constant on the createTasks function.

NOTE: You do not need to download the key, behind the scene Cloud Tasks will take care of this via IAM, cool eh?

IAM Roles per function

Ok y'all, we are almost there. One feature we really need asap is to be able to apply roles per function, (Hint: The Serverless Framework handle this very nice, at least on AWS 🤷🏻‍♂️)

Add Role to the Service Account

ok back to the CLI, type this:

Note: Make sure you change the values with your data:

  • preOrder: Name of the HTTP Function
  • serviceAccount: The email for the service account you just created on the prev step
  • region: The Cloud Function Region
gcloud beta functions add-iam-policy-binding preOrder --member serviceAccount:http-function-invoker-only@cloud-tasks-sample-263214.iam.gserviceaccount.com --role roles/cloudfunctions.invoker --region us-central1

After this, you will be prompted with a message which contains something like this:

bindings:
- members:
  - allUsers
  - YOUR SERVICE ACCOUNT

Now we need to remove the allUsers role:

Again y'all, remember to change preOrder for your HTTP function name.

gcloud beta functions remove-iam-policy-binding preOrder --member allUsers --role roles/cloudfunctions.invoker --region us-central1

NICE! so if everything when great, we can just test trying to hit the endpoint from the outside world

http-post

Voila! you are done! this will get you started with Cloud Tasks like a pro.

Next Steps

Remember to subscribe to the newslatter and share this if you like it so I will know if someone is actually reading this or give up and move on with my life!

The repo is here.

Peace!

Enjoyed this post? Receive the next one in your inbox!

I hand pick all the best resources about Firebase and GCP around the web.


Not bullshit, not spam, just good content, promised 😘.


Reddit
Linkedin