Oct 5 2009

Google Summer of Code successfully finished

Hi !

I finished Google Summer of Code 2009 successfully, and I guess I already spent all the money I received from Google :) Anyway my dissertation was accepted by my University and I want to give a special thank to Philip Johnson and Austen Ito from University of Hawaii for this great experience in the world of open source software. Those guys are racks. To every student willing to participate in GSOC: i can just recommend it !

At the moment I am looking for a job out there, and while I’m waiting for replies I keep myself busy with a little project. An idea I had some month ago but no time because of studies. The idea is actually huge and i am looking at its feasibility. It may becomes reality, PUSH THUMBS !

Harvey


Aug 18 2009

GSoC Hackystat Weekly Update 13

Hi folks !

a small update this week, while i have to sprint to the finish line

Last week:

Some more documentation and implementation. I had some problems with the CBR solutions which where not mapped trough hibernate in JCOLIBRi, thus my cases had no solution. I could fix the problem by changing the hibernate configuration in the source code of JCOLIBRi. Therefor integration into IVY RoundUp is not possible and I will integrate the library in my source code. I also started implementing the overall MAPE cycle for each service in the Hackystat Service Manager. This will be documented by the end of the week in a developer guide in the wiki of the project:

This week:

Finishing up the tasks i described for the future in my last blog post.


Aug 11 2009

GSoC Hackystat Weekly Update 12

Hi Everybody !

Overview
This blog entry is finally an interesting one. Remember to the overall picture:
The overall concept was described in my post of the GSoC Weekly Update 3 of the 8th of June . The only conceptual adoption is that there is always just one single agent for each service with monitoring and execution functionality.

Hackystat Service Manager

As already mentioned in earlier posts, the basic framework was already implemented end of July. The basic system consists of a simple Graphical User Interface and the agents. The agents are SOA services implemented in REST, and are only capable of starting and stopping the services so far (Execution part of agents).

Last week I was able to finish the set up of the blue “CBR” component. I used the Jcolibri2 CBR framework for this. JColibri is accessing an embedded Apache Derby database in the Hackystat Service Manager. I set up some initial cases with the CBR features I created over the last weeks in the database. Now one can create a new CBR Problem description and send it to the JColibri framework which returns a set of cases that are similar to the CBR problem description.

The CBR component:

The documentation and introduction to JColibri can be found here: JCOLIBRi Documentation

1. First we have to set up an embedded Apache Derby database, with a table structure as follows:

create table case_base (caseId varchar(15), service varchar(50), reachable integer,
reqLatency real, adeqHeapsize integer, exceHeapSize integer,
dbLatency real, solutionCode integer

The table structure depends on the CBR features, which are extendable and the table structure can be changed at any time in my Java Class called CaseBaseInitializer. The same class allows to enter some initial cases into the database or, later, load an exported database for porting capabilities.

2. A class called CaseRecommender implements the StandardCBRAppication interface of the JColibri library. This is the core class that creates a connection the database via hibernate and maps the cases to objects. Therefore we have four XML configuration files in the directory of the application called:

  • databaseconfig.xml
  • hibernate.cfg.xml
  • CaseSolution.hbm.xml
  • CaseDescription.hbm.xml

For details on the concept and configuration files, look at the JColibri documentation.

3. The CaseRecommender class maps the columns of the Database (CBR-features) with Similiarity Functionalities.
That looks like this:

// First configure the NN scoring
NNConfig simConfig = new NNConfig();

//Set Global similarity function to average
simConfig.setDescriptionSimFunction(new Average());

//Create attributes for Mapping CBR features
Attribute service = new Attribute("service", CaseDescription.class);
Attribute reachable = new Attribute("reachable", CaseDescription.class);
Attribute reqLatency = new Attribute("reqLatency", CaseDescription.class);
Attribute adeqHeapsize = new Attribute("adeqHeapsize", CaseDescription.class);
Attribute exceHeapsize = new Attribute("exceHeapsize", CaseDescription.class);
Attribute dbLatency = new Attribute("dbLatency", CaseDescription.class);

//Set similarity function for each CBR feature
simConfig.addMapping(service, new Equal());
//simConfig.setWeight(service, 0.5);
simConfig.addMapping(reachable, new Equal());
simConfig.addMapping(reqLatency, new Interval(100));
simConfig.addMapping(adeqHeapsize, new Interval(4000));
simConfig.addMapping(exceHeapsize, new Interval(4000));
simConfig.addMapping(dbLatency, new Interval(100));

4. Now a new CaseDescription Object can be created and send to the CBR application to retrieve the most similar cases.

This is all working successfully, what it needs is some optimization and perfection of the CBR features and cases as well as the implementation of the monitoring and execution.

What work is left ?

In the next two weeks I will complete the system, by:

  • implementing the monitoring functions for the CBR features described here: http://www.weitz.lu/cbrfeatures_draft.pdf
  • creating the Analysis component, if some metrics show misbehavior the metrics are captured and send to the CBR JColibri framework
  • creating the Plan component will adopt the solution of the most similar case and execute it. Retrieve the solution, Reuse the solution, Revise the solution, Retain the solution. The 4 major CBR steps.
  • adding execution fucntionalities to the Agents for fixing the misbehavior appropriate to the case solution.

Aug 4 2009

GSoC Hackystat Weekly Update 11

Last week:

First of all I did some more documentation and research for my dissertation. I would say that I have more than a third of my dissertation done. It is mostly the background and introduction part that is finished. I have also a solid amount of literature and notes for the main part of the dissertation which I have to transpose.

I did some adoption to my projects for the Continuous Integration in Hackystat. I used the hackystat-developer-example as template and copied the Apache ANT files. I adopted the build.xml in order to use the IVY Roundup libraries.

I am exploring a concept for the realization of CBR, where the similarity measures and best case retrieval is done with myCRB, unfortunately myCBR can only be integrated to a Tomcat web server. This is why I will also explore JColibri .

Next week:

Google Summer Code finishes the 25th of August, thus I will focus again on integrating CBR into my system and improving the system and the graphical user interface. This involves:
Implementing monitoring features into the agents in order to collect metrics for the CBR features. (Monitoring)

  • Detecting misbehavior out of the metrics (Analysis)
  • Capturing the metrics and create a specific case
  • Sending the case to the CBR framework (myCBR/JColibri) and retrieving the most similar case (Planing)
  • Looking up the solution for this case, every solution has a unique number and a java class called solution finder will implement the solution. Here I will probably go for Java Reflection API in order to write to new solutions and load them into the solution finder. This is just a thought so far, more on the “execution-phase” later.
  • Hervé


    Jul 28 2009

    GSoC Hackystat Weekly Update 10

    Last week:

    More documentation, i created also a section that is an introduction to Hackystat, when its finished i will also produce a ten page long PDF out of it, maybe there is some interest.

    I found a cool Java library called JAMon (http://jamonapi.sourceforge.net/) to monitor Java Applications and JDBC. I think the JDBC monitoring could be useful for Sensorbase, while the Java monitoring expects to run the application as Java Object. So far the HSM runs and stops the service over Java Processbuilder, but I will investigate if there is another integration method. Anyway I keep on looking for JVM Monitoring tools: like Java VisualVM (In which one cannot access the internals), but i found out that it is based on jvmstats.
    Philip pointed my in an e-mail to Epsilon (http://www.obix-labs.com/dr/bin/view/Epsilon/), but for me the website is not reachable since weeks.

    I started a thread in the mailing list on the Hackystat CBR cases, and I started to analyze and write down how the ideas in the mailing-list could be implemented. Therefore I create a little PDF (http://www.weitz.lu/cbrfeatures_draft.pdf) initializing the list of features that each service provide. It is to implement these features as metrics in the monitoring agents of the HSM and to expand the list.

    This week:

    Finishing the first three chapters of the documentation (Introduction, Background (Software architecture, SOA, REST, Hackystat) and “Autonomous service oriented computing” (CBR)). Thus “Autonomous service oriented computing in Hackystat”, still to go. Expanding the CBR feature list and play around with its implementation on the monitoring level and think about the cases and their implementation.

    Greetings
    Harvey


    Jul 20 2009

    GSoC Hackystat Weekly Update 9

    Hi folks !

    Last week:
    I did some write up for my dissertation. I integrated the missing libraries into my project, I tried out some ways to achieve this and came across the Fatjar Eclipse plugin (http://fjep.sourceforge.net/), which is really easy to use and efficient. Fatjar just integrates the jars that are set in the java build path of the eclipse project. I came up with some CBR features, mainly gained trough the REST API of the different services. On top of these CBR features I am doing some research on JVM and Java Application Monitoring tools. There is a hell of a lot out there and I try to find an open source solution or a library that suits my need and fits with my current system to gain more CBR features. I also looked at the mail about standardization of the Hackystat projects and hope to achieve this by the end of the month, if I get it right its about Check-styles and IVY. Since I got only the REST and the JGoodies Forms libraries I think I will get there quick.

    This week:
    More documentation :) as my supervisor from my University requested, to avoid pressure at the end of the summer and more CBR features (Investigating Monitoring tools) and some cases hopefully by the end of the week. I read trough some tutorials and papers about MyCBR, but this is still work in progress and I cant see away yet to integrate it. I might also look at JColibri CBR Framework in the hope that it is supporting my understanding and maybe easier to integrate. And as soon as I have some solid documentation it will be integrated partly into my wiki.

    Cheers, have a nice week everybody

    Hervé


    Jul 14 2009

    GSoC Hackystat Weekly Update 8

    I am glad to announce that I passed the Google Summer of Code Midterm evaluation as one can read in the blog of Philip Johnson (http://philipmjohnson.blogspot.com/2009/07/google-summer-of-code-midterm-results.html)

    Last Week:

    I finished the basic system that can run and stop the Hackystat services, in an architecture that is service oriented, flexible and extendable, so that the services can be controlled wherever they are geographically located and new services can be added easily. The REST API and functionality is limited to start and stopping as well as monitoring the accessibility of the services. The first system release with its documentation is available at Google Code Hosting (http://code.google.com/p/hackystat-service-manager/)

    This Week:

    The REST API has to be extended and new monitoring metrics have to be found that will also create the so called “features” of the Cased-Based Reasoning for achieving the autonomy of the service oriented architecture. Therefore I will also work with myCBR (www.myCBR-project.net) for prototyping the CBR system . The CBR system will be integrated in the framework.
    First priority is updating my dissertation and doing more research on CBR with a focus on Hackystat.

    Regards,

    Hervé


    Jul 7 2009

    GSoC Hackystat Weekly Update 7

    Hello, everybody

    Last week I was working on the basic framework, I  made some advances with JSW, anyhow the whole progress can be found in the Hackystat Mailing list. I worked on implementing the REST services (Agents) and I discovered after some investigation two good CBR frameworks: JColibri (http://sourceforge.net/projects/jcolibri-cbr/) and myCRB (http://www.myCBR.com).

    This week I will keep on fighting on the two fronts:
    1) Working on the framework, I played around with YAJSW as JSW alternative but wasn’t successful yet. I downloaded it, set the wrapper.jar on my project classpath and followed the step of “Starting a Java application” in the documentation (http://yajsw.sourceforge.com)
    2) I will investigate the two CBR tools and I will write a little overview and summary about them and try to prototype some cases with the tools. They are both open source so I look for an API or any access to internals, so that I could use it in some way for the framework CBR component.

    Regards,

    Harvey


    Jun 29 2009

    GSoC Hackystat Weekly Update 6

    Hello,

    To be honest I wasn’t that productive over last week, the weather was terrible nice here in Luxembourg which doesn’t happen that often. At least I’m fine with the REST API concept and i was able to implement the different services (execution and monitoring), but I decided to merge them for each service. In the theory it sounds great to separate the two agents but on programming and usage level it is more comfortable to have just one agent for monitoring and execution. I started to define the REST API which is very depending on the cases. I looked at the different REST API Specification of the different services and at the Java client to get a good idea for the cases. In CBR features should have two possible values, like true or false. These features, in there different states, create a case. One could use all important REST API methods to create those features but I really think here i need some interaction with experienced users. I was talking with Austen about this a while ago, but any input is welcome, i might open a thread for this in the mailing list soon.

    I also googled for some CBR tools and some papers describing or evaluating CBR tools, this is my track besides the programming (Launching and Monitoring)

    The weather is getting worse again, and I can turn back to the desk and accomplish some work this week. The only thing where I’m really stuck right now is JSW (using JSW successfully from code not from the command line) and I was investing several hours without any productivity. I wrote Austen an E-Mail about that.

    For this week and the next week I hope that I get the basic framework up and running and a good REST API depending on the features of the CBR cases for the monitoring agent.


    Jun 22 2009

    GSoC Hackystat Weekly Update 5

    Hello everybody, here is my weekly dissertation report:

    Last Week:

    I made some progress with JSW (The mac OSX 64.bit problem is solved). I guess I am quite close the only thing right now is to integrate the wrapper.conf for the JVM configuration on code level. It seems to me when using the wrapper in the console, scripts bind this file to the wrapper program. I saw lines like: wrapper-exe d- /conf/wrapper.conf, and looking at the testwrapper.sh script, the path to the wrapper.conf is also provided in there. Thus it is investigate how these configuration parameters can be set on code level, since I used the open source project 3.2.5 and the wrapper.jar in latest binary distribution 3.3.5 to solve the mac OSX problem.

    I familiarized myself with REST for the execution agent and monitoring agent, which went quite well. I also started to define the REST API for the agents, but so far just the very obvious methods like Start, Stop for execution and getStatus (implemented so far with the “ping” REST API definition of the services) for the monitoring agent.

    So far I have a little UI for the framework showing the Hackystat Services with buttons to start and stop them and a label with there status. Thus the structure is there (framework with agents), just the JSW implementation is not working correctly yet.

    I was also looking at the REST API of each service in depth to figure out what can be monitored. I have to find a way to put this into proper structured CBR cases. My supervisor from my University also pointed me in the direction of learning adaption knowledge to improve case-based reasoning. I was reading papers about that topic.

    This week I try to finish some CBR cases and try to advance with JSW and API of the agents.

    For the midterm evaluation, I hope that the basic system for launching and monitoring the services is build so that I can come to the important part (autonomic computing :)) in the beginning of July. A lot of the theory is already there, so that I can try out some concepts by extending the basic system I described above.