Messages in the Cloud Pub/Sub format need to be exported to various formats for applications to consume them or import them into another data source. The following section explains the various options available for exporting a message in various formats. Create a new Pub/Sub topic in your project for this demo and add a subscriber to that topic. Note down the topic name and subscriber name for use in the following command. This demo uses pubsubTopic as its topic name and pubSubTopicSubscriber as its topic subscriber name.
Run the following command to publish a message to the intended topic in Google Cloud Pub/Sub. Replace Message Text and attribute in the message as per your project:
gcloud beta pubsub topics publish pubsubTopic --message "Message Text" --attribute="EmployeeID=20,FirstName=Vinod,LastName=Kumar,DateOfJoining=2016-02-16,Country=Norway"
The preceding command generates a message ID and displays it as shown here:
Use the following command to pull the message without the --auto-ack flag to see the contents of the message in JSON format in the console. Replace pubSubTopicSubscriber with the Pub/Sub subscriber of your project. The message will be displayed in JSON format. The JSON format inserts each attribute in a new line; so it cannot be directly imported to BigQuery:
gcloud beta pubsub subscriptions pull pubSubTopicSubscriber --format=json
The preceding command displays the message with its attributes in JSON format:
Run the following command to see the output flattened, where each key-value pair is written in one line. Replace pubSubTopicSubscriber with the Pub/Sub subscriber of your project:
gcloud beta pubsub subscriptions pull pubSubTopicSubscriber --format=flattened
The preceding command produces an output as shown in the following screenshot. The message and its attributes are displayed in each row:
The following command exports the message in CSV format for the message that was published earlier. CSV files require a projection to be specified, which is a defining list of attributes from a message to be exported to the file. The following code will export all the columns in the message data. The columns are specified in the format option within (). Remove [no-heading] if you want a heading to be added to the output:
gcloud beta pubsub subscriptions pull pubSubTopicSubscriber --format="csv[no-heading](message.attributes.EmployeeID,message.attributes.FirstName,message.attributes.LastName,message.attributes.DateOfJoining,message.attributes.Country)"
The preceding command displays the specified message attributes delimited by ,:
To know the list of formats that can be used to export the Cloud Pub/Sub message, run the following command:
gcloud topic formats