Posts

Spark Codes

Clustering import org.apache.spark.mllib.clustering.KMeans import org.apache.spark.mllib.linalg.Vectors val data = sc.textFile("data.csv") val parsedData = data.map(s => Vectors.dense(s.split(',').map(_.toDouble))) val numClusters = 6 val numIterations = 300 val clusters = KMeans.train(parsedData, numClusters, numIterations) val WSSSE = clusters.computeCost(parsedData) println("Within Set Sum of Squared Errors = " + WSSSE) val labeledVectors = clusters.predict(parsedData) labelVectors.saveAsTextFile val centers = clusters.clusterCenters =================================================================== scala> textFile.count() // Number of items in this RDD res0: Long = 126 scala> textFile.first() // First item in this RDD res1: String = # Apache Spark scala> val linesWithSpark = textFile.filter(line => line.contains("Spark")) linesWithSpark: spark.RDD[String] = spark.FilteredRDD@7dd4af09 scala> te...

Building Search Engine like Google

Interesting Reads: http://www.rose-hulman.edu/~bryan/googleFinalVersionFixed.pdf

Machine Intelligence / Learning Landscape

Image

Raspberry Pi

Static IP for Wifi sudo vi /etc/network/interfaces Add wifi string ID (Different from wifi SSID) auto lo iface lo inet loopback iface eth0 inet dhcp allow-hotplug wlan0 iface wlan0 inet manual wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf iface default inet dhcp iface home_static inet static    address 192.168.0.53    netmask 255.255.255.0    gateway 192.168.0.1  

Perl

Part 1  Recap of Datastructures  Complex Data structures Anon CDS -Hashes Part 2  Regular Expressions  Subroutines & Files Part 3  Modules  CPAN/DBI/CGI/N-w ------------------------------------------------------- Perl identify the DS by the prefix symbol  Scalar - $  - a value  List   - ()  Array  - @  Hashes - % define a lexical scalar -  my $a; default value scalar    -  undef how to check for undef  -  if(defined($a)){ } get input from keyboard -  $a=<STDIN>    # \n stops                            @arr=<STDIN>  # EOF stops output to console       -  print STDOUT "Hello";                            print "Hello"; Errors                  -  print S...

Unix and Hadoop

SCP to a seperate PORT scp -P 5050 asd.tar.gz user@192.168.1.15:/home/user Tomcat Webapps Location /usr/share/tomcat/webapps Get IP address ifconfig eth0 | awk '/inet /{print $2}' Get Tomcat logs tail -f /usr/share/apache-tomcat-7.0.30/logs/catalina.out tail -f -n 10 Running Hadoop Jobs set mapred.job.queue.name=dev hadoop jar acs.jar -D mapreduce.reduce.speculative=false -D mapreduce.map.speculative=false -D mapred.job.queue.name=dev -D mapreduce.task.io.sort.factor=256 -D file.pattern=.*20110110.* -D mapred.reduce.slowstart.completed.maps=0.9 -D mapred.reduce.tasks=10 /Input /Output hadoop jar test.jar -D mapred.job.queue.name=dev -D mapred.textoutputformat.separator=, Glassfish Webapps folder location /usr/share/glassfish4/glassfish/domains/domain1/applications Killing Mysql Processes ps -ef | grep mysql | awk -F" " '{system("kill -9  "$2)}' Starting MySQL /etc/init.d/mysqld start Mo...

Free Dashboards

Image
http://webdesign.tutsplus.com/tutorials/build-a-dynamic-dashboard-with-chartjs--webdesign-14363 http://keen.github.io/dashboards/examples/ http://usebootstrap.com/theme/sb-admin