SVN Commands

* To Commit only particular Revision of a file

svn update -r37325 /appl/abc/xyz/bvp_udf.jar
svn update -r37374 /appl/abc/xyz/ki/efg/run.sh

sqoop 2

Errors

1) Error Message
ava.lang.RuntimeException:

java.lang.RuntimeException :com.mysql.jdbc.exceptions.jdbc4 .CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received

any packets from the server.at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164 )

at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)

at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:606)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)

at org.apache.hadoop.mapred.Child$4.run(Child.java:270)

at java.security.AccessController.doPrivilege(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)

at org.apache.hadoop.mapred.Child.main(Child.java:264) 

• Problem: Communications Link Failure caused by incorrect permissions.

•Solution:

–Verify that you can connect to the database from the node where you are running Sqoop:

•$ mysql --host= --database=test --user= --password

–Add the network port for the server to your my.cnf file

–Set up a user account to connect via Sqoop

. Grant permissions to the user to access the database over the network:

• Log into MySQL as root

mysql-u root -p

•Issue the following command:

mysql> grant all privileges on test.* to 'testuser'@'%' identified by 'testpassword' 

Create a Proxy for the mysql host and port on the current host if not present already

mysql-proxy --proxy-backend-addresses=qa-srv:3308 --log-level=debug --proxy-address=127.0.0.1:3306

try login on another console using proxy now

 mysql -u  -h 127.0.0.1 

Alternate to this is another Tool used on the Linux is 

Rinetd – redirects TCP connections from one IP address and port to another

 

edit  /etc/rinetd.conf and add the remote and local server binding

# bindadress bindport connectaddress connectport
192.168.2.1 80 192.168.2.3 80
192.168.2.1 443 192.168.2.3 443

Sqoop 1

Sqoop

• Sqoop from local system i.e. system on which the database as well as hadoop runs.
• Sqoop from remote system, here we need to grant the permission so the communication link failure and other issues are avoided as the data will be access over the network, so the database admin needs to provide permission where the data can access over the network using a particular ip address.

Sqoop -> Hive -> Indexing
Schemaless storage of the data specially data coming from XML and JSON objects.
Zookeeper