Cloud Functions and Redis

Reddit
Linkedin

so you can sleep well without thinking that your firebase invoice is going to be huge

Since the first time I heard about Firebase, I've always seemed one or two (baaa actually a shit ton of) people complaining about possible getting a million dollars invoice if they get a DOS or doing magic to come out with data structure models to avoid that extra read and keep the costs low af cause serverless should tend to free (yeah sure), anyway, I've always mention to boys and girls that caching is your friend, you need to sleep well and you don't need that negativity in your life. 

In case you don't want to keep reading, the repo is here and the video tutorial is here.

Cloud MemoryStore 

Whatever as a Service (trademark pending) is on fire these days and our old friend Redis is part of the party! I'm not going to waste your time explaining what it is, you can always check this, the long story made short, is a key-value store and GPC offers a ready to use solution called Cloud Memory .

A little of story first

It was not possible to integrate Cloud Functions with MemoryStore in the past because the way the sandbox works exposing the instance, basically you get an instance placed on your VPC and connect to it via the internal network traffic, all that sadness is still there but now we have Serverless VPC Access which works as a proxy between the Functions and the MemoryStore instances, end of the story.

The Architecture

Let's see how to integrate with cloud function, our goal is to cache our request to Firestore check if memory and return the value if it exists, otherwise, we go query it and save it for the next request to use.

diagram

In order to do this, we have to:

  • Enable Serverless VPC Access.
  • Create the connector.
  • Create the Redis instance.
  • Give the Cloud Function User the right permissions.
  • Deploy the Cloud Functions.

Enabling Services

Go to Apis and Services and enable Serverless VPC Service 

enable-vpc-gif

Now let's create the connector, it will work as our proxy between the functions and the vpc world, not only for Redis instance but also for any resources we have into our vpc(s) It's important to mention that the Redis instance and the Cloud Functions need to be on the same Region, in this case us-central and you need to select the range of IP addresses for your instance(s) to be placed on, in this case 10.3.0.0/28 and finally, the vpc where you want to use, in the case we are using the default, you can also set the minimum and maximum throughput, since we don't have any requirement I will use the default for the exercise. 

create-connector-gif

Let's create the Redis instance using Cloud Memorystore , after this, take note of the region and the IP address of the instance, you will need them later.

create-redis-instance-gif

Now we need to ensure we have enough permission to execute operations from our Cloud Function , new need to add Compute Network User and Serverless VPC Access User to our Cloud Functions User , you can always create a Role instead ;-)

iam-permission-gif

Finally, we are ready to code our functions :)

The Function

We need to install the redis package

npm install redis --save

The idea is as explained in the diagram, we are moving the request to a Cloud Function I'm using an HTTP fn but you can use a callable or even a background function if your goal is to access the record from a background process, we check first on the cache memory, if we find the record we are done, otherwise, we retrieve it from firestore and save it on the cache for the next query to use it.

import * as functions from 'firebase-functions'
import * as admin from 'firebase-admin'

admin.initializeApp()

const REDIS_PORT = 6379
const REDIS_HOST = `10.0.0.3`

const firestore = admin.firestore()
const ordersRef = firestore.collection(`orders`)

const redisLib = require('redis')
const client = redisLib.createClient(REDIS_PORT, REDIS_HOST)

const fromCache = (key: string): Promise<string | null> => new Promise(async resolve => {
  if (client.connected === false) {
    resolve(null) 
  } else {
    return await client.get(key, function(_err, reply) {
      resolve(reply)
    })
  }
})

export const redis = functions.https.onRequest(async (req, res) => {
  const id = req.query.id
  let response: string | null = await fromCache(id)
  if (response) {
    res.status(200).send(Object.assign({}, JSON.parse(response), { from: 'redis' }))
  } else {
    let record = await ordersRef.doc(id).get().then(doc => doc.exists ? doc.data() : null)
    if (record) {
      if (client.connected === true) {
        await client.set(id, JSON.stringify(record))
      }
      res.status(200).send(Object.assign({}, record, { from: 'firestore' }))
    } else {
      res.status(404).send({ message: `not found`})
    }
  }
})

As you can see, only the first query will pay for caching the data. Now we are in full in sync, let's test it! I've extended the object response to include the source, so we can validate they come from the right place. You can always get more fancy on the fromCache function, this just serve the example :), notice we are also checking that we can connect to the instance, otherwise, it will throw a connection error.

firebase deploy --only functions

Set the connector

We are not done! we need to link the connector to the Function (yeah I would love to have infrastructure as code in here, post about it coming soon ;-))

link-connector

Now we can test!

first-test

What about changes?

Depend on your use case, you use case, you can possible flush the cache once a day, hour, a week if your information doesn't change that much but if you need full sync 24/7, we can use the same pattern we used on the algolia tutorial.

export const cacheUpdate = functions
  .firestore.document(`orders/{doc}`).onWrite(async (change, _context) => {
    const oldData = change.before
    const newData = change.after
    const data = newData.data()
    const id = newData.id

    if (!oldData.exists && newData.exists) {
        // creating
         await client.set(id, JSON.stringify(data))
         return Promise.resolve(true)
      } else if (!newData.exists && oldData.exists) {
        // deleting
        await client.del(id)
        return Promise.resolve(true)
      } else  {
        // updating
        await client.set(id, JSON.stringify(data))
        return Promise.resolve(true)
    }
})

Remember to link the connector for this function as well!

test-changes

That's it! now you have a couple of caching strategies to add to your belt! Use it wisely.

Remember to subscribe so my mom can be proud!

Check the video if you are more of a visual person (I talk about the costs).

Resources

Enjoyed this post? Receive the next one in your inbox!

I hand pick all the best resources about Firebase and GCP around the web.


Not bullshit, not spam, just good content, promised 😘.


Reddit
Linkedin