Data exchange with [EX|IM]PORT

When working on data migration or release deployment, we may need to move data between different environments or clusters. In HQL, EXPORT and IMPORT statements are available to move data between HDFS in different environments or clusters. The EXPORT statement exports both data and metadata from a table or partition. Metadata is exported in a file called _metadata. Data is exported in a subdirectory called data, as follows:

> EXPORT TABLE employee TO '/tmp/output5';
No rows affected (0.19 seconds)

> dfs -ls -R /tmp/output5/;
+--------------------------------+
| DFS Output |
+--------------------------------+
| ... /tmp/output5/_metadata |
| ... /tmp/output5/data |
| ... /tmp/output5/data/000000_0 |
+--------------------------------+
3 rows selected (0.014 seconds)
For EXPORT, the database name can be used before the table name without any syntax error, but the database is useless and ignored by the IMPORT statement.

Once exported, we can copy exported files to other clusters manually or with the hadoop distcp <srcurl> <desturl> command. Then, we can import data in the following ways:

  1. Import data into a new table. If the table exists, it will throw an error:
      > IMPORT TABLE FROM '/tmp/output5'; -- By default, use exported 
name
Error: Error while compiling statement: FAILED: SemanticException
[Error 10119]: Table exists and contains data files
(state=42000,code=10119)


> IMPORT TABLE empolyee_imported -- Specify a table imported
> FROM '/tmp/output5';
No rows affected (0.788 seconds)
  1. Import data to an external table, where the LOCATION property is optional:
      > IMPORT EXTERNAL TABLE empolyee_imported_external
> FROM '/tmp/output5'
> LOCATION '/tmp/output6';
No rows affected (0.256 seconds)
  1. Export and import partitions:
      > EXPORT TABLE employee_partitioned partition
> (year=2018, month=12) TO '/tmp/output7';
No rows affected (0.247 seconds)

> IMPORT TABLE employee_partitioned_imported
> FROM '/tmp/output7';
No rows affected (0.14 seconds)
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.23.130.191