Review Board 1.7.22


BigDecimals converted to String in inconsistent format

Review Request #9081 - Created Jan. 24, 2013 and submitted

David Robson
SQOOP-830
Reviewers
Sqoop
sqoop-trunk
Currently when BigDecimal fields are saved as Strings Sqoop uses the ToString method. This leads to values like "0.0000001" being stored as "1E-7" which doesn't seem ideal.
This patch changes Sqoop to use ToPlainString for BigDecimals so they will always be stored in the same format. This should have minimal effect as they can still be converted back to BigDecimals no matter which way they are stored - and the scale doesn't seem relevant - it seems to always be zero anyway so there shouldn't be any change there.
I added a new parameter "sqoop.bigdecimal.format.string" which can be set to false to revert to the old behaviour.
I didn't add this as a command line parameter as it seems like something most users would not change so didn't want to confuse the user with another option - they can override it in sqoop-site.xml or on the command line using -D.
Have manually tested text file, avro file and hbase imports using both values of the new parameter.
Also checked that if the parameter is not set it will use the toPlainString.
I tested sequence files but there is no change as they don't use the toString methods.
src/docs/user/import.txt
Revision 82e74dd New Change
[20] 572 lines
[+20]
573
    --class-name SomeTableType
573
    --class-name SomeTableType
574
----
574
----
575

    
   
575

   
576
This command will load the +SomeTableType+ class out of +mydatatypes.jar+.
576
This command will load the +SomeTableType+ class out of +mydatatypes.jar+.
577

    
   
577

   

    
   
578
Additional Import Configuration Properties

    
   
579
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    
   
580
There are some additional properties which can be configured by modifying

    
   
581
+conf/sqoop-site.xml+. Properties can be specified the same as in Hadoop

    
   
582
configuration files, for example:

    
   
583

   

    
   
584
----

    
   
585
  <property>

    
   
586
    <name>property.name</name>

    
   
587
    <value>property.value</value>

    
   
588
  </property>

    
   
589
----

    
   
590

   

    
   
591
They can also be specified on the command line in the generic arguments, for

    
   
592
example:

    
   
593

   

    
   
594
----

    
   
595
sqoop import -D property.name=property.value ...

    
   
596
----

    
   
597

   

    
   
598
.Additional import configuration properties:

    
   
599
[grid="all"]

    
   
600
`-------------------------------------`----------------------------------------

    
   
601
Argument                               Description

    
   
602
-------------------------------------------------------------------------------

    
   
603
+sqoop.bigdecimal.format.string+       Controls how BigDecimal columns will   \

    
   
604
                                       formatted when stored as a String. A   \

    
   
605
                                       value of +true+ (default) will use     \

    
   
606
                                       toPlainString to store them without an \

    
   
607
                                       exponent component (0.0000001); while  \

    
   
608
                                       a value of +false+ will use toString   \

    
   
609
                                       which may include an exponent (1E-7)

    
   
610
-------------------------------------------------------------------------------

    
   
611

   
578

    
   
612

   
579
Example Invocations
613
Example Invocations
580
~~~~~~~~~~~~~~~~~~~
614
~~~~~~~~~~~~~~~~~~~
581

    
   
615

   
582
The following examples illustrate how to use the import tool in a variety
616
The following examples illustrate how to use the import tool in a variety
[+20] [20] 108 lines
src/java/org/apache/sqoop/hbase/HBasePutProcessor.java
Revision 64a1d18 New Change
 
src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java
Revision 1f52ba9 New Change
 
src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java
Revision 30db288 New Change
 
src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
Revision f6e2e72 New Change
 
src/java/org/apache/sqoop/orm/ClassWriter.java
Revision 126b406 New Change
 
src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java
Revision f9370c4 New Change
 
src/test/org/apache/sqoop/TestBigDecimalExport.java
New File
 
src/test/org/apache/sqoop/TestBigDecimalImport.java
New File
 
  1. src/docs/user/import.txt: Loading...
  2. src/java/org/apache/sqoop/hbase/HBasePutProcessor.java: Loading...
  3. src/java/org/apache/sqoop/hbase/ToStringPutTransformer.java: Loading...
  4. src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java: Loading...
  5. src/java/org/apache/sqoop/mapreduce/ImportJobBase.java: Loading...
  6. src/java/org/apache/sqoop/orm/ClassWriter.java: Loading...
  7. src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java: Loading...
  8. src/test/org/apache/sqoop/TestBigDecimalExport.java: Loading...
  9. src/test/org/apache/sqoop/TestBigDecimalImport.java: Loading...