Follow Along with Instructions using Client Libraries

To follow along within your own Google Cloud Project, follow the directions below.

1. Create a Cloud Platform project

2. Enable billing for your project

3. Enable the Google Cloud Storage API

4. Set up authentication with a service account so you can access the API from your local workstation

   In the Cloud Console, go to the IAM & Admin > Service Accounts

   From the Service account list, select New service account.

   In the Service account name field, enter a name.

   From the Role list, select Project > Owner.

   1. In a production environment, you would want to go with the least privilege, and you do not want to assign a service account Project Owner.

Click Create. A JSON file that contains your key downloads to your computer.

We will use this JSON to authorize our sample code access to the Cloud Storage API.

Create a Bucket

The below code provides a sample of how to use the Node.js cloud storage client library to create a bucket. It will create a bucket in the location that the developer defines. In this code snippet, the bucket name is defined as “my-bucket” (Line 1).

const bucketName = 'my-bucket';
// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');
// Creates the client
const storage = new Storage();
async function createBucket() {
 const [bucket] = await storage.createBucket(bucketName, {
   location: 'US-CENTRAL1',
   regional: true,
 });
 console.log(`Bucket ${bucket.name} created.`);
}
createBucket().catch(console.error);

Starting on line 7, you will notice the second argument is an object that contains metadata that we are passing to the storage client which tells the storage client to create our bucket in the respective location. You can also pass the object as follows:

const bucketName = 'my-bucket';
// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');
// Creates the client
const storage = new Storage();
const metadata = {
 location: 'US-CENTRAL1',
 regional: true
};
async function createBucket() {
 const [bucket] = await storage.createBucket(bucketName, metadata);
 console.log(`Bucket ${bucket.name} created.`);
}
createBucket().catch(console.error);

In this code block, we create a variable line 6 to hold the parameters we want to pass to the storage client. Line 11 now includes the metadata object created on line 6 which will be passed to our create bucket request. Below is a table of the metadata options you can use during the create bucket request.

One important option when creating the bucket is choosing the class as well as if it will be multiRegional. Let’s review the storage classes available for Google Cloud as this will be an important choice from a cost perspective and will also make sure you’re choosing the appropriate class for your data access profile.

The table below was taken from the Google Cloud storage documentation. I would recommend checking the Google Cloud documentation to confirm none of the SLAs have changed since the published date of the book.

The minimum storage duration is a key metric to consider when choosing the class. Note, all classes perform the same from a latency perspective; what you need to consider is the access profile. In our framework, our files will be accessed fairly often because we’re providing users the ability to play the audio file. Due to its access profile, the standard class for our framework meets our requirements and will be the most effective. Google Cloud will charge you an early delete fee if the file is deleted before it has been stored for the duration of the class.

Let’s go ahead and create our first bucket using Node.js.

1. On your local workstation, git clone the following repository:
   https://gitlab.com/building-cloud-native-applications-google-cloud/chapter-two.git

2. From the cloned repository, go to the chapter-two folder.

3. Open the createBucket.js file in your favorite editor.

4. On line 6, replace the “key.json” with the service account key you downloaded.

5. On line 7, replace “ruicosta-blog” with your project id.

6. On the terminal within this folder, run npm install

7. On the terminal, run node createBucket.js

8. On successful execution, it should look like Figure 2-6
   

Figure 2-9. Code Screenshot for createBucket.js

You can go to the Google Cloud Storage console and your newly created bucket should be listed. Congratulations! You programmatically created your first bucket.

Create a Bucket with Bucket Locks

A bucket lock allows you to create a retention policy that locks the data preventing it from being deleted or overwritten. You can also lock a retention policy. Once it is locked you cannot unlock it; you will only be able to increase the retention period. See the warning in Figure 2-10 from the Cloud Console. This is warning you that if you lock the retention policy which is set for 5 days, you will lose the ability to shorten or remove it. If you want to delete the objects or the bucket itself, you can only do so after the 5 days have expired.

Figure 2-10. Locking the Cloud Storage Policy

If you attempt to delete objects younger than the retention period it will result in a PERMISSION_DENIED error.

Let’s programmatically create a retention policy for 6 months. Objects in this bucket will not be able to be overwritten or deleted. Review line 5. This is the location you can set the duration of the bucket lock.

// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');
// Creates a client from a Google service account key.
const storage = new Storage({
 keyFilename: "key.json",
 projectId: "ruicosta-blog"
});
const retentionPeriod = 15780000; // 6 months.
const bucketName = 'rui-costa-bucket-name-2020-01';
async function createBucket() {
 await storage
 .createBucket(bucketName)
 .setRetentionPeriod(retentionPeriod);
 console.log(`Bucket ${bucketName} created.`);
}
createBucket().catch(console.error);

To run the code snippet locally on your desktop follow these steps:

1. If you cloned the repo before, you can skip this step: On your local workstation git clone the following repository:
   https://gitlab.com/building-cloud-native-applications-google-cloud/chapter-two.git

2. From the cloned repository, go to the chapter-two folder.

3. Open the createBucketLock.js file in your favorite editor.

4. On line 5 replace the “key.json” with the service account key you downloaded.

5. On line 6 replace “ruicosta-blog” with your project id.

6. If you run npm install in a previous step within this folder you can skip this step: On the terminal within this folder run npm install

7. On the terminal run node createBucketLock.js

8. On successful execution, it should look like Figure 2-11

Figure 2-11. Code Screenshot for createBucketLock.js

You created a new bucket with a retention lock period of 6 months. If you try to upload a new file to the bucket and delete the uploaded file you will receive an error message.

List Buckets

Let’s now list the buckets we programmatically created. This is a useful command as it allows you to list all the buckets in your project, which can allow you to perform actions like Data Loss Prevention. We can loop through all the buckets and objects, then scan them for PII information with the Google Cloud Data Loss Prevention API. Line 12 we use a forEach() which calls the provided callback function once for each element in an array. Each element in the array is our list of buckets within our project.

// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');
// Creates a client
const storage = new Storage({
 keyFilename: "key.json",
 projectId: "ruicosta-blog"
});
async function listBuckets() {
 // Lists all buckets in the current project
 const [buckets] = await storage.getBuckets();
 console.log('Buckets:');
 buckets.forEach(bucket => {
   console.log(bucket.name);
 });
}
listBuckets().catch(console.error);

To run the code snippet locally on your desktop follow these steps:

1. If you cloned the repo before, you can skip this step: On your local workstation git clone the following repository:
   https://gitlab.com/building-cloud-native-applications-google-cloud/chapter-two.git

2. From the cloned repository, go to the chapter-two folder.

3. Open the listBuckets.js file in your favorite editor.

4. On line 5 replace the “key.json” with the service account key you downloaded.

5. On line 6 replace “ruicosta-blog” with your project id.

6. If you run npm install in a previous step within this folder you can skip this step: On the terminal within this folder run npm install

7. On the terminal run node listBuckets.js

8. On successful execution, it should look like Figure 2-12

Figure 2-12. Code Screenshot for listBuckes.js

Upload a File

We created a bucket. Now, getting objects to the bucket is the funniest part. In the code snippet below we will upload a file called “rui_costa.jpg” to cloud storage within the bucket named “rui-costa-bucket-name-2020”.

const bucketName = 'rui-costa-bucket-name-2020';
const filename = 'rui_costa.jpg';
// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');
// Creates a client
const storage = new Storage({
   keyFilename: 'key.json',
   projectId: 'ruicosta-blog'
 });
async function uploadFile() {
 // Uploads a local file to the bucket
 await storage.bucket(bucketName).upload(filename, {
   metadata: {
     cacheControl: 'public, max-age=31536000',
   },
 });
 console.log(`${filename} uploaded to ${bucketName}.`);
}
uploadFile().catch(console.error);

To run the code snippet locally on your desktop follow these steps:

1. If you cloned the repo before, you can skip this step: On your local workstation git clone the following repository:
   https://gitlab.com/building-cloud-native-applications-google-cloud/chapter-two.git

2. From the cloned repository, go to the chapter-two folder.

3. Open the uploadFile.js file in your favorite editor.

4. On line 1 replace with your bucket name

5. On line 2 replace with the file you want to upload

6. On line 7 replace the “key.json” with the service account key you downloaded.

7. On line 8 replace “ruicosta-blog” with your project id.

8. If you run npm install in a previous step within this folder you can skip this step: On the terminal within this folder run npm install

9. On the terminal run node uploadFile.js

10. On successful execution, it should look like Figure 2-10

Figure 2-13. Code Screenshot for uploadFile.js

That was easy! We now programmatically uploaded a file to our bucket.

Download the File

Another option you might need is getting data out of Cloud Storage. We created buckets and uploaded a file but how can we download this file now? The code snippet below will download the file you just uploaded to the local location you define. On line 3 you can change the name of the file that will be saved locally so as to not overwrite the existing file we uploaded.

const bucketName = 'rui-costa-bucket-name-2020';
const srcFilename = 'rui_costa.jpg';
const destFilename = 'download_rui_costa.jpg';
// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');
// Creates a client
const storage = new Storage({
   keyFilename: 'key.json',
   projectId: 'ruicosta-blog'
 });
async function downloadFile() {
 const options = {
   destination: destFilename,
 };
 // Downloads the file
 await storage.bucket(bucketName).file(srcFilename).download(options);
 console.log(
   `gs://${bucketName}/${srcFilename} downloaded to ${destFilename}.`
 );
}
downloadFile().catch(console.error);

To run the code snippet locally on your desktop follow these steps:

1. If you cloned the repo before, you can skip this step. Otherwise, on your local workstation git clone the following repository: https://gitlab.com/building-cloud-native-applications-google-cloud/chapter-two.git

2. From the cloned repository, go to the Chapter 2 folder.

3. Open the downloadFile.js file in your favorite editor.

4. On line 1 replace with your bucket name

5. On line 2 replace with the file you want to download

6. On line 3 replace with the name you want to save the download file locally

7. On line 8 replace the “key.json” with the service account key you downloaded.

8. On line 9 replace “ruicosta-blog” with your project id.

9. If you run npm install in a previous step within this folder you can skip this step: On the terminal within this folder run npm install

10. On the terminal run node downloadFile.js

11. On successful execution, it should look like Figure 2-14

Figure 2-14. Code Screenshot for downloadFile.js

Now we are having fun. We created a bucket, uploaded a file, and then downloaded the file! Congratulations.

Signed URL with Python

So far we have focused on the Node.js client library, but we can perform all the actions we have done thus far with other languages. Let’s say I wanted to send a file that was existing in Cloud Storage to an endpoint. One way we can accomplish this is via the signed URL method available to us. Example: We want to send a PDF file to the endpoint for the endpoint to convert the PDF to a PNG file. Below is a sample code that shows us how to use signed URLs in Python.

Signed URLs is a URL that provides access to users and applications for a limited time. The signed URL allows users to access the object without authentication. In our framework, we authenticate and authorize the user via Identity Aware Proxy. At this point we generate a signed URL for a limited time that allows the user to play the respective audio file which is an object with Cloud Storage.

from google.cloud import storage
storage_client = storage.Client.from_service_account_json(service_acct)
bucket = storage_client.bucket(bucket_name)
for blob in bucket.list_blobs(prefix=str(prefix + "/")):
 if blob.name.endswith(".png"):
   url_lifetime = 3600  # Seconds in an hour
   serving_url = blob.generate_signed_url(expiration=url_lifetime, version='v4')

Let’s review the code snippet. On line 5, we have a for loop that iterates through the defined bucket name on line 3. For each object that ends with the extension of “.png” we generate a signed URL with the expiration time set to 3600 seconds. The signed URL could be passed as a parameter to an endpoint. Since the URL has the needed authentication and authorization for the object, the endpoint will not be required to authenticate for the set period of time. Google makes it pretty easy for us to generate signed URLs that allow users or applications to access our objects without requiring them to authenticate or authorize.

Cloud Function Libraries

Cloud Functions provides you the ability to use external modules. The dependencies for our framework are managed with npm and defined in a file called package.json. This allows developers to be creative with their functions. Our framework leverages the following external modules:

const PubSub = require(`@google-cloud/pubsub`);
const storage = require('@google-cloud/storage')();
const speech = require('@google-cloud/speech').v1p1beta1;
const client = new speech.SpeechClient();
const uniqid = require('uniqid');
const ffmpeg = require('fluent-ffmpeg');
const ffmpegPath = require('@ffmpeg-installer/ffmpeg').path;

The fluent-ffmpeg and ffmpeg-installer/ffmpeg allow our function to check the codec and length of the audio file. When submitting a job to Speech-to-Text for transcription we need to configure the sampleRateHertz parameter. Using the external libraries, we extract this information and pass it to the Speech-to-Text API. We are using the long-running job method available from the Speech-to-Text API. When we submit an audio file to be transcribed for a long-running job we need to wait for the API to complete, this process can take minutes to hours. The formula used is n/2 for how long the API will take to complete. For this we need to wait a certain amount of time before we check to see if the API is done.

If we continuously poll the API we might hit a quota limit with Google Cloud. To avoid hitting a quota we only check the API for a completed job after the duration of the audio file divided by two is completed. Example: If the audio file is 1 hour long we will only check the API for completion after 30 minutes have passed. We will cover this in-depth in the Dataflow chapter as here we only submit the job to Speech-to-Text and receive the job name which then is sent as a key:value pair to Pub/Sub.

GitLab CI/CD and Cloud Functions

Let’s practice deploying a Cloud Function with GitLab CI/CD.

We leverage a core component of Cloud-native applications Continuous Integration (CI) and Continuous Delivery (CD) architecture to automate deployment tasks, and perform peer reviews prior to pushing the code to production. For our example we will leverage GitLab for Source Code Management and CI/CD.

For our first integration step with GitLab https://about.gitlab.com/, we will keep it simple and focus on getting our code developed and deployed faster. You will need a GitLab account to follow along. To authorize GitLab access to our Google Cloud project we will need to create a service account and assign the required roles. Always keeping in mind to work with least privilege methodology.

1. Open the IAM & Admin in the Google Cloud Console.

   

2. On the menu click Service Accounts

   

3. Click CREATE SERVICE ACCOUNT

   

4. Enter your Service account details:

   Service Account Name

   Service Account ID

   Service Account Description

5. Click Create.

6. Assign the following roles to the Service Account

   Cloud Functions Admin

   Service Account User

7. Click Continue

8. Click Done

9. Locate the newly created Service Account, click on the actions icon and select create key.

   

10. Choose JSON and click CREATE. This will download the JSON file to your local workstation.

11. Head over to your GitLab Project, and under settings go to CI/CD

12. Click Expand Variables

13. Create two new variables labeled as:

    PROJECT_ID

    SERVICE_ACCOUNT

14. Enter your Google Cloud project id

15. Open the JSON file for the service account you downloaded before, copy and paste it’s content to the SERVICE_ACCOUNT key. It should look something like Figure 2-15:

Figure 2-15. GitLab Variables Screenshot

At this point you have created a service account and provided the service account details to your GitLab project which will allow GitLab CI/CD to deploy the Cloud Function to your Google Cloud project. To trigger a pipeline we will need to create a .gitlab-ci.yml file, this file defines the order of the pipeline. Below is our .gitlab-ci.yml file we will use to deploy our cloud function.

image: google/cloud-sdk:latest
deploy_production:
 stage: deploy
 only:
   - master
 script:
   - echo $SERVICE_ACCOUNT > ${HOME}/gcloud-service-key.json
   - gcloud auth activate-service-account --key-file ${HOME}/gcloud-service-key.json
   - gcloud --quiet --project $PROJECT_ID functions deploy safLongRunJobFunc --runtime=nodejs8 --trigger-event=google.storage.object.finalize --trigger-resource=audio-uploads-42892

Pipeline configuration begins with jobs. Jobs are the most fundamental element of a configuration file. In our configuration file our job name is deploy_production. You will notice in Figure 2-16, the name displayed for our job.

Figure 2-16. Screenshot of GitLab Pipeline

The script keyword is a required component of the configuration file. It is the shell script which is executed by the runner. Here we are authenticating gcloud with the service account created and assigned to the variable within GitLab. The next step is the deployment of Cloud Functions using the gcloud command. We are also using the google/cloud-sdk:latest docker image as it contains the gcloud command, allowing for a lightweight docker image during the deployment.

If you have not cloned the GitLab repository, go ahead and clone it to your local workstation. Open your favorite IDE, for this exercise I will be using Visual Code. Download or clone the following repository:

https://gitlab.com/building-cloud-native-applications-google-cloud/speech-analysis-framework.git

Copy the files in the saf-longrun-job-func folder to your GitLab repository local folder. For this you will need to have created a repository in GitLab and cloned the empty repository. It should look like Figure 2-17

Figure 2-17. Screenshot of the sample code for the Cloud Function

1. Create a new branch

Figure 2-18. Creating a Feature Branch in Visual Studio Code

1. Make a small change to the index.js file, commit the changes, and push the branch to the origin of your repository.

2. Back in GitLab console create a merge request

   

   You should be able to see the merge request. At this point, the CI/CD pipeline will not be executed. Our pipeline will only be triggered when it is merged into the master branch. This allows reviewers to review the changes and accept the merge request. This was defined in our GitLab configuration file as:

only:
   - master

After reviewing the changes you can Merge the request.

1. Head over to the CI/CD and view the jobs, the job should now be running:

You have successfully created a GitLab pipeline that only executes when merged to the master branch. Developers can work off feature branches and only allowed users can merge the request to the master branch which will then execute the GitLab CI/CD pipeline which in our framework will deploy the source code to Google Cloud Function.