Hadoop Meet-up - Large Scale Graph Processing On HBase and Map/Reduce on Greenplum

I attended the Hadoop Meet-up on Tuesday titled "Large Scale Graph Processing On HBase and Map/Reduce on Greenplum". You can view the event here.

I attended this event for two reasons: It has been a couple of years since I worked intimately with Hadoop and I wanted to see how others are using it. I was also hoping the discussion on Large Scale Graph Processing would be useful for my research.

Though the presentations were somewhat stimulating, I didn't find much I could use for my work.

Hadoop + SPARQL

I began working with cloud computing about 2 years ago because of my interest in using this environment for semantic web applications. It seems to be of interest to others too.

https://opencirrus.org/content/sparql-query-over-hadoop-very-large-rdf-dataset
http://portal.acm.org/citation.cfm?id=1779599.1779605
http://cs264.org/projects/web/Rohloff_Kurt/rohloff/index.html

Notes on XMPP Prototype

I've been recently investigating XMPP for Cloud Computing. After reading the specifications and learning basically what it is all about, I've begun working on a prototype that would accomplish similar tasks that I accomplish with mesh4x . Basically the goal is offline/online synchronization. So far, I've played around with Jabber, tigase, and a couple of other servers. I need basically server and client support. So far, I like OpenFire and Smack. Right now I have code written in Java that basically creates a chat between two users. I created two accounts at jabber.iitsp.com and wrote some basic code. Now, my two clients are communicating by chatting with one another.

My next steps:

1. Use OpenFire locally as the server
2. Write code that will do what I do with mesh4x
3. Hook it into Google Maps

Basic code:

XMPPConnection connection = new XMPPConnection("jabber.iitsp.com");
connection.connect();
connection.login(userName, password);
Chat chat = connection.getChatManager().createChat(person, new MessageListener() {

public void processMessage(Chat chat, Message message) {
System.out.println("Received message: " + message.getBody());
}
});

chat.sendMessage("Hi there!");

I do a bit more in my code to keep the thread running and to recognize who the user is and respond with a customized message but this is the code that actually sends and receives the message. This is taken from the Smack documentation as a basic test. If you want to just test it one way, create an account at Trillian using XMPP as the protocol with one of your user accounts: user1@jabber.iitsp.com. Then set up your Java code as user2 who wants to send to user1. Your trillian client will pop up your message.

Oracle SqlPlus Error

A remedy to the following error:
ORA-01034: Oracle not available
...shared memory realm does not exist
...

Execute the following commands:
sqlplus /nolog
connect / as sysdba
startup

I have found Oracle on Ubuntu to be a bit hacky. I've been tweaking configuration each day to get a stable instance running. This wasn't a clean install and Oracle states that Ubuntu is an unsupported os. However, with a bit of tweaking and using the directions I provided in a previous post, one can get a fairly stable Oracle instance running.

Testing CouchDB

If you made it through the installation process and successfully started CouchDB then you could use your browser to get familiar with CouchDB or you could use curl.

I used curl because it is command line driven, fast and gets the job done.

Test access:
curl http://localhost:5984/

A response of {"couchdb": "Welcome", "version": "0.8.0-incubating"} will indicate we are ready.

If you do not get this response then check to make sure Couchdb is running.

You can also go to the browser to double check: http://localhost:5984.

Creating a database:
curl -X PUT http://localhost:5984/mydemodb/

A repsonse of {"ok",true} will indicate that the database was created. If you get an illegal name error then check your database name. Is it all lower case, no spaces, no special characters?

Deleting a database:
curl -X DELETE http://localhost:5984/mydemodb/

Adding data to the database:

Presuming a database was created, create a testdata.dat file with your data. Here is an example:

{
"field1": "value1",
"field2": "value2"
}

Enter the command to save the data:
curl -T testdata.dat -X POST http://localhost:5984/mydemodb/

With a response of:
{"ok": true, "id": "...", "rev":"..."}

Installing Oracle on Ubuntu

Ubuntu isn't a supported OS for Oracle so the installation process can be a bit of a challenge. However, I have found a fairly decent reference for exactly what to tweak in order to successfully install Oracle.

This article provides step by step instruction on installing Oracle on Ubuntu. The only additional notes I would add are the following:

1. You will need to export your display before installing.
2. Ignore the glib errors that occur during the installation process.

Enjoy....

CouchDB

If you are struggling with the installation of CouchDB on Ubuntu, particularly the dreaded SpiderMonkey issue. The process outlined at http://barkingiguana.com/2008/06/28/installing-couchdb-080-on-ubuntu-804 worked perfectly. Blow away your current CouchDB directory first, then follow the process outlined at http://barkingiguana.com/2008/06/28/installing-couchdb-080-on-ubuntu-804 and you will quickly discover success.