lucene-bytebuffer

What is lucene-bytebuffer.

lucene-bytebuffer is Lucene Directory implementation using Direct ByteBuffer. Directory in lucene is backing storage for index. Lucene uses directory for storing index contents. So there is RAMDirectory, FileDirectory, MemoryMappedFileDirectory, NIODirectory each presenting various different options. lucene-bytebuffer will allow in-memory index to grow upto several gigabytes without incurring garbage collection cost.

Mostly indexes are 90 to 95% read and 2-5% write ie. index hardly changes. If index is huge it will cost a lot in terms Garbage Collection CPU cycles. RAMDirectory holds arrays of size 1024 so for 1GB index its 1 million array objects. So as size gets increased in-memory index performance degrades due to garbage collection.

What if you want to index say 5GB data? Use off-heap bytebuffer backed directory.

Another question is why would you want to use lucene in-memory indexing. May be as Cache which can be queried on more than one property of object indexed?

BigMemory - Memory with no garbage collection overhead How?

I just came to know about BigMemory. After reading description and Terracotta CTO Ari's blog I wondered how does it work? BigMemory stores object in memoey without any garbage collection tax and stores it in native memory. It means you are no longer bound by JVM heap limit and you can use all memory (which is dirt cheap) available. And all this in Plain Java. No JNI.

I mean how did it happened that If there was technique available why cant any available cache framework used it? By implementing the this technique along with complete memory manager, Terracotta thus showed that its indeed leader in distributed cache. Why spent effort in optimizing garbage collection times which is a black art.

Lets assume that We have such mechanism which allows us to store java objects in native memory. So how do we implement BigMemory like system. Let's focus just on Cache usecase . You need to track objects in cache. But Object tracking is quite easy in cache usecase. They are removed directly by cache eviction threads or user so ultimately its map.remove(key).

So how does BigMemory work?. This is just my guess. It may be using Direct ByteBuffers. I did some googling around allocating native memory in JVM and found this article. and Yes Direct ByteBuffers are stored in Native memory. So below is my little shot at mimicking BigMemory. Basically each object is converted in ByteBuffer and stored in ByteBuffer. Some time and memory can further be saved by using faster and more smart serialization techniques.

public Object put(K key, V value){
  try {
   ByteArrayOutputStream baos = new ByteArrayOutputStream();
   ObjectOutputStream stream  = new ObjectOutputStream(baos);
   stream.writeObject(value);
   stream.close();
   byte b[] = baos.toByteArray();
   ByteBuffer buffer = ByteBuffer.allocateDirect(b.length);
   ByteBuffer buffer2 = map.put(key, buffer);
   if(buffer2!=null){
    ByteArrayInputStream bais = new ByteArrayInputStream(buffer2.array());  
    ObjectInputStream oois = new ObjectInputStream(bais);
    V object = (V)oois.readObject();
    oois.close();
    return object;
   }
   else return null;   
  } catch (IOException e) {
   e.printStackTrace();
   throw new RuntimeException(e);
  } catch (ClassNotFoundException e) {
   e.printStackTrace();
   throw new RuntimeException(e);
  }
 }
 
 public V get(Object key){
  try{
   ByteBuffer buffer2 = map.get(key);
   if(buffer2!=null){
    ByteArrayInputStream bais = new ByteArrayInputStream(buffer2.array());  
    ObjectInputStream oois = new ObjectInputStream(bais);
    V object = (V)oois.readObject();
    oois.close();
    return object;
   }
   else return null;
  }catch (IOException e) {
   e.printStackTrace();
   throw new RuntimeException(e);
  } catch (ClassNotFoundException e) {
   e.printStackTrace();
   throw new RuntimeException(e);
  } 
 }

The solution above stores keys in JVM heap and objects on non-heap memory. So If number of objects are 2M, JVM only has overhead of 2M objects while doing GC and not 4M ( in case of Map and 4-5M in case of EHCACHE where object is serialized and wrapped in ehcache Element object). This can be even further reduced by implementing some sort of hashing. Key is hashed into one of the buckets which are serialized and stored in buffers. You de-serialise bucket metadata and then iterate though it to find out actual buffer and then locate element. But BigMemory is just not cache its entire object graph storage.

Thus version presented here is poor man's BigMemory compared to what BigMemory will have but it still can save hugh GC tuning efforts in cases where there is hugh memory requirement in the application and thus heavy garbage collection. But Terracotta's BigMemory would be with much more smart strategies and data structures, integrated with distributed garbage collector and implementing all cluster semantics as all other terracotta clustered data structure do. Waiting for integration of BigMemory into Terracotta server. Memory has been one of the biggest paint points in Terracotta. That would give Terracotta unlimited-scalability in terms of number of distributed objects stored.

Querying Java Objects stored in Terracotta's NAM Part 3

First and Second part of this series I talked about Querying data structures available in Java. First part specifically talked about existing ones like JoSQL, JxPath and Quaere and discussed indexing problem. Second part specifically talked about lucene and jofti as indexing frameworks and wrote small framework to test and get some performance numbers. This test showed that B-tree index (memory and possibly disk based) is suitable and not lucene index. Third part of this series now I am discussing my own small framework- tcquerymap which is rewrite of Jofti. Jofti is old and not maintained and uses jdk14 libraries. Now even documentation link does not work : http://prism-index.com/guide.html.

querymap documentation link : Getting Started With Querymap
querymap source download : Complete Source Code with Eclipse Project
querymap TIM : Copy this TIM to terracotta modules directory to use it
querymap terracotta sample app : Download sample eclipse App. You can use Terracotta eclipse plugin to launch it within eclipse

So the basic idea is maintaining in-memory indexes which index String Ids with "Comparable" as Keys. All primitive wrappers in Java implement this interface so no special conversion is needed.


How it works
It scans java objects and makes comparable objects for each of the property mentioned to be indexed. It inserts these comparable objects in its own tree against String Ids assigned to the java object. So in-effect its nothing more than maintaining many in-memory Maps. In fact the implementation that I wrote uses JDK TreeMap and not b-tree map. But in-future it will be replaced by more performant B-tree.

Querying
One needs to understand that with in-memory object indexes its only possible to implement subset of SQL query and no join queries. To start with, small framework only implements following operations : arithmetic operands : <,>,<=,>=,==. Logical operands : AND , OR. Range : BETWEEN, Set : IN

To implement querying SQL parser needs to be implemented. I choose to avoid writing parser and implemented direct API kind of querying similar to Quaere. I find Domain specific languages more intuitive since when developer writes code for it he know what querying he writes. So interface looks like below. It is not as good as quaere. Quaere is DSL, below API is just few interfaces implemented. Here execute returns List of IDs put into index against the properties.


import static com.google.code.querymap.ObjectQuery.*;

Collection col = from(Domain.class).where(
gt("inner.property2", 60),
lt("inner.property2",89)
eq("inner.property3",random.nextInt(NUM_OBJECTs))
).execute();



So basically it is equivalent of follwing SQL

select from Domain
where inner.property2 > 60
and inner.property2 < property3=rand(NUM_OBJECTS)


How it performs
Naturally is not as performant as Jofti since it used JDK tree map which uses red-black tree. In future when I complete writing my own b-tree implementation I expect to perform as good as Jofti. Jofti further implemented node level locking so multiple concurrent insert opertations can work parellel. This can also be implemented too. But query performance is not bad and I expect it to improve with b-tree implementation.


Integration with Terracotta
Since it uses TreeMap it is cluster-able easily. Attached tc-config.xml has all correct declarations for it. One more advantage is with Terracotta is that object identifier are readily generated by Terracotta. See implementation TerracottaQueryMap. Please dont compare performance of TerracottaQueryMap against HashMap, CHM or Terracotta Distributed Map(Concurrent String Map old name) since all these are just single index maps so easy to stripe or employ mulitple locks.

For using it as TIM you need to add following lines to tc-config.xml


<modules>
<module id="com.google.code" name="querymap-1.0" version="1.0.0">
</module>


In code you can use it as queryable map as follows. Here propList is list of properties to be indexed.


TerracottaQueryMap map = new TerracottaQueryMap(Domain.class,proplist));

map.put(key,domain1);
map.put(key,domain2);
map.put(key,domain3);


To Query.


Collection col =map.entrySet(
from(Domain.class).
where(
eq("inner.property2",random.nextInt(NUM_OBJECTs))
)
);


Future directions planned
Java has hugh limitation for memory intensive application. So achieve scale two approaches : partition index and merge results or use disk to overflow index pages. Other thing I see can be implemented is that when query selects random elements these random elements need to be faulted from Terracotta server thus degrading performance, same elements can be read from local store too like EHCACHE. Later on this topic separately.

If you think this framework is useful please let me know. You can download its source as Eclipse project(sole dependency on tc.jar) and as Terracotta Integration Module here.

Tushar

Erlang and Concurrency

Here I write a lot about my experiments with Terracotta which is shared durable memory for Java applications which are multi-threaded. But some days before I came across erlang, language which is created at Erricsson for running fault-tolerant telecom applications. Whats so special about it?. If you google or rather "Bing" it you will find plenty of things about it. How cool and scalable it is?. If you read this you will even come to know How Goldman Sachs was using it to gain significant advantage in program trading over other competitors and how others are trying to steal leaked source code.

So erlang is not procedural programming language its functional programming language. Frankly I also need to understand whats so different about it. But I saw this presentation of infoQ site about erlang concurrency and was amazed. Right from starting I always thought about following graph - throughput increases as a function of in-coming request rate till some point but after it stabilizes and then it drops. It drops because of system overload. In a perfectly cpu-intensive lock-contention free application this will happen because of cpu context switching.


But in erlang it stays constant instead your latency (response time) increases. This is according to Little's Law. Little law says relation between throughput and latency is number of users in system. N = RX.




Above is famous graph of benchmark of YAWS (Http server written in erlang) against Apache and you can see how early apache gets saturated and dies. You can read details here but explanation is given here is :

"The problem with Apache is not related to the Apache code per se but is due to the manner in which the underlying operating system (Linux) implements concurrency. We believe that any system implemented using operating system threads and processes would exhibit similar performance. Erlang does not make use of the underlying OS's threads and processes for managing its own process pool and thus does not suffer from these limitations."


So basically all magic is erlang's concurrency model : No Shared State, Only message passing between light-weight processes. Erlang processes are way lighter than Java Threads since they are logical entities and not tied to user-level or kernel-threads. Thus erlang shows "No Shared State" concurrency model scales well. Since now JVM is touted as platform, I am looking forward to see erlang implementation on JVM and see how it does against other concurrent interpreted languages - scala. This is great post why JVM is unfit for such porting. May be Java 8 ( I think closures are not part of Java 7). This is also interesting read about Erlang on Java : Erlang Concurrency model on JVM . Some of work on writing OTP(erlang's sdk for writing applications) for scala http://github.com/jboner/scala-otp/tree/master/.

I have already got Programming Erlang book now looking forward to write first program in OTP.

Tushar

Links : Java Sample Apps

Many times you hear or read something cool about some framework or tool and you are interested in Sample application written for it just to play with it, browse and go through source-code to find out how to code it.

Here is list of great sample Apps that I just stumbled upon while reading this great blog about Tomcat Clustering.

Link : http://www.mediafire.com/jbs-blog-examples

List is
* IBM-DB2-JDBC-XML.zip
* IBM-DB2-JDBC-Relational.zip
* Google-App-Engine.zip
* WebServices-JAX-WS-Java-SE.zip
* RESTful-WebServices-Apache-CXF.zip
* WebServices-Apache-CXF-Spring-2.5.zip
* WebServices-Apache-CXF.zip
* WebServices-WSIT-Reliable-Messaging.zip
* WebServices-JAX-WS-Web-App-Client-Basic-Security.zip
* WebServices-JAX-WS-Web-App-Client.zip
* WebServices-JAX-WS-Web-App.zip
* PMD-Clover2-Cobertura-Maven2-Test.zip
* WebServices-JAX-WS.zip
* WebServices-Axis2-with-Eclipse-client.zip
* Spring-Maven2-annotations-example.zip
* Spring-Maven2-basic-example.zip

I will add following to above lists which I know about

Terracotta Samples Application written by Team

• Terracotta Reference Application : Examinator
  


• 
  Terracotta with Spring and EHCACHE
  


• 
  Terracotta, Hibernate, Spring
  


• 
  Terracotta, Spring, Hibernate
  


• Sample Web App with Maven, Terracotta
  

Terracotta's Hibernate Integration

This post is re-post of my earlier write-behind post but in different perspective : Terracotta's Hibernate integration 3.1

With version 3.1 Terracotta has implemented its own Caching for Hibernate Second Level Caching Provider. Earlier Terracotta's hibernate integration approach was : clustering EHCACHE. Terracotta with its JVM clustering ability, it was easily possible to cluster any POJO structure. So before 3.1, you might have used EHCACHE as hibernate second level cahce provider and tim-hibernate and tim-ehcache for clustering second level cache. With version 3.1 onwards terracotta will have its own cache backed by map-evictor and concurrent string map. Apart from this new hibernate integration has lots of new additions like cache admin console and read-write cache. Cache is always up-to-date and coherent.

But what I feel is that Terracotta platform is way more capable and following additional features can be added to make applications more scalable. These are just cool ideas.

Cache Warm-up feature
It would be nice feature to refresh or load cache whenever application or application cluster is starting up. This can easily be implemented with some sort of CacheLoader interface where Terracotta can callback this interface when faulting cache objects from terracotta server during first access. But such warm-up is only required on full cluster restart otherwise lot of meaningful cache entrites will get overwritten.

Write-Behind Caching
When you think of cache you will arrive at these cache strategies : Read-Through Caching, Write-Through Caching, Write-Behind Caching. Hibernate Second Level cache is Read-Write-Through Cache where if cache miss occurs, entity is read from database and then handed over to cache for susequent access. But H2LC is not Write-Behind caching. With Terracotta's disk persistence and asynchronsous module it would be really efficient for certain use-cases to implement write-behind. Currently hibernate just directly writes to database. Instead if its modified to write to second level cache and persistent async-database-queue, this would decrease latency and increase throughput dramatically. Imagine if you can schedule all your database writes in non-business hours using tim-async. I find write-behind is certainly the best way to reduce pressure on database. And with Terracotta's clusterwide coherent persistent datastore its practicaly possible. Terracotta would be your database guard taking all your querying as well as database inserts on its shoulders.
But this model would require certain changes in the way hibernate works. especially query cache. Since now Terracotta will have latest snapshot of yor System of Record, queries have to be executed against cache and not database. Thus it can not be generic solution. You can implemented write-behind only in certain cases where your business use case permits it. On the other hand to solve query problem Querymap that i disucssed in my previous posts can be used to query certain type of data. So if your business use case permits write-behind and query-map can give you very fast database accelerator. In one of my previous jobs I was working on financial application where certain set of objects were modified at very high rate and same were queried against. For such application classic replicated H2LC does not bring any value, instead it will degrade the performance due to overhead during frequent-cluster-wide updates. But Terracotta will make it scalable, forwarding updates only to Node on which cache entry exists, updating the object clusterwide so when AsyncProcessor picks it up it will contain all the changes made. Its Terracotta's DSO Magic.

Advantage here is that you dont have to do religious shift of Killing Your Database Totally. Database is your System of Record. With Terracotta Hibernate Accerlerator you are only delaying updates to SOR and not replacing it.

Currently I am going through Hibernate source code and learning how hiberante event mechanism works. My guess is that write-behind can be implemented with hibernate events. If not I may try to modify the source code to add write-behind and h2lc-cache querying capability. Hibernate search is similar where instead of classic session you get Indexing-aware session.

With Terracotta FX (assuming your application requires more than 4000 write operations per second - avg throughput of one un-tuned Terracotta server) your write throughput will increase linearly which is not possible with any RDBMS on any type of hardware.

I hope Terracotta will add these features in coming versions. Terracotta 3.1 Hibernate Integration is just start.

Querying Java Objects stored in Terracotta's NAM Part 2

First part of this series, I talked about existing frameworks and what I found out is that they lack indexing hence rarely useful for large data-sets. So I tried finding out how to do indexing. My idea was simple : index objects and store reference to object in index then with Terracotta you can cluster objects and index as well. So it becomes "queryable" datastore. My first attempt was to find out how indexing is done. By book it says b-tree index. I found out this(jdbm) framework which is trying to do the persistent DB in Java by implementing B-tree indexes on disk.I took only b tree and implemented simple query parser. What it does is that it traverses b-tree and finds out tuples and then returns them.

After this first attempt, then I experimented with Lucene. Lucene is not tree-index, its inverted index but it has lot of capability and its fast, supports in-memory and disk-based indexes.

So here is my little framework for queryable datastore :



 public interface TCQueryMap extends Map{
  void init();
  Map query(String query);
 }




Naturally its extension of Map which is single index. My implementation wraps a HashMap with ReadWrite locks and Lucene RAMDirectory index. So all get/puts hit index within lock boundaries and then you can query index. This is very simple, I have not gone into complexities like spill-over of index onto disk etc.



 LuceneIndexingConfig config = new LuceneIndexingConfig();
 List propList = new ArrayList();
 // index three properties only
 propList.add("accountName");
 propList.add("person.age");
 propList.add("person.name.firstName");
 
 config.setIndexPropertyList(propList);
 
 TCQueryMap indexer = new LuceneQueryStore(config);
 
 // add object
 Acccount account .....
 indexer.put("user99",account);
  
 // query object 
 Map col = indexer.query("person.age:21");




I also came to know about Jofti from one of comments. Jotfi is what I would eventually like to write. I don't know why its not used by many people. One reason could be its not maintained. I found it pretty useful so I plugged in Jofti as well in my framework.

Now lets compare it with simple Hibernate-JDBC based solution. Obviously its not perfect. SQL is way more complex and expressive language. But here we are talking about cached data and I am sure once data is cached ( it means objectified from join query on relational DB) very few times you will require join, its mostly "where clause" of one or more conditions. Lucene does that very well.

So lets see numbers. I have not done any tuning apart from standard lucene stuff. One of main parameters is how many properties you want to index. This determines index size, memory and speed.

Below is small benchmark showing 60K objects inserts with three properties indexed and then random queries on three properties

Lucene Inserts/Sec = 1000
Jofti Inserts/Sec = 8793.78
Lucene Queries/Sec = 5172
Jofti Queries/Sec =13636.36

Results for 14 properties indexed :

Lucene Inserts/Sec = 740
Jofti Inserts/Sec = 4866.96
Lucene Queries/Sec = 3750
Jofti Queries/Sec = 12500

Since Jofti is Tree index it outperforms Lucene Index. The problem with Lucene is that once index gets bigger insert performance slows down. Also these numbers are taken with one commit on one put operation. If you index lot of objects together and then commit, lucene is also fast, that's how it is to be used - Batch API. On the other hand Jotfi is fast, I could not find any details about being thread-safe and other concurrency issues so I wrapped it around Lock. I don't know why Jofti is not used by many people.

Also what if you can run Hibernate/JPA queries on Map? that would be great. Its already done by hibernate team. They run query against Second level cache but it would big task to find out and extract idea out of it. Just a thought. Second Level Query cache gets invalidated when you modify single entity, imagine if we update the same object in QueryMap cache you dont need Query cache of course querying capability is not great.

Another thought that comes in my mind is clustering in-memory databases like H2 or HSQLDB. Imagine the benefits of it. But then its anti-terracotta. Why? it would be Relational DB with baggage of ORM mismatch.

Entire source code you can download it from here : http://code.google.com/tc-querymap/. Tar file is just bunch of java files and its very early
prototype. Stay tuned to project, I will update it once I finish with proper integration with Terracotta.

So if you find it useful please leave comments, I would love to hear from you.

Links : Distributed Hash Tables

Nice List of Distributed Key-Value Stores : http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/

Querying Java Objects stored in Terracotta's NAM

This post is inspired from : http://forums.terracotta.org/forums/posts/list/1965.page . Nothing new .. just another word in blogosphere.

Terracotta is gr8 clustering solution in-fact its platform-level service hence has large number of uses. One of the use is using it as database. Terracotta can never replace database but it can play role of data storage media very well. One of major disadvantage is lack of querying data. Only way you can query data is Map. Map is like single index so if you want to get list of object satisfying some criteria you are required to integrate through entire collection. There are already APIs written for querying java collections. So you can use them with Terracotta NAM.

When you think about querying there are lot of factors : Query Language, Its Performance - Optimizers, Operations Supported : Select, Update, Delete, Joins

JoSQL

JoSQL is good API for querying java collections with good documentation. I did small test with 1 million objects and random query took around 800ms which is way too much. Again its simple iteration through collection due to lack of indexes and query execution plan. Problem with Indexes is that Object graphs can change and at every change you are required to recompute the index which would be difficult to do : as complex as Terracotta's bytecode instrumentation. You can find test code here

Query Language : Moderately good, Performance : Not good for large collections, No update or delete only select projection queries. No joins

Quaere

Quaere is a very flexible DSL that lets you perform a wide range of queries against any data structure that is an array, or implements the java.lang.Iterable. Its sort of port of LINQ of .Net world. I think linq is next generation data quering tool -cleaner. Quaere is not query language but query API just like Hiberate Criteria query but more elegant. I really liked quaere - its really powerful its support join operation. You can read this post for details : Solving Puzzles with Quaere Its still beta level and not released. One of Queare's another sister project is its JPA integration. Imagine you could write standard JPA application with Quaere as query language and Terracotta as persistent store. No need of database. But as with JoSQL Quaere is also slow. I mean slower than RDBMS. I did small test with 1 million object similar to JoSQL test and response time was similar to JoSQL. You can download test code here

This post also discusses jmap's OQL implementation. It uses rhino javascript engine behind with hashtables.I did consider it to port for Terracotta but its custom written for Object Heap Dumps. JxPath is another tool with which you can query java collections using XPath expressions. I did not evaluate JxPath since i felt it will be on similar lines of JoSQL and Quaere, only different flavor. If you have used XPath earlier then this is much easier to use.

GlazedList is event driven list API specially designed for Swing Applications displaying table and list data. But if you consider List of Objects as table (each object is row and its properties as columns) a proper in-memory index can be maintained for querying. But this applies to only root object level. What if inner object in object graph store in your container changes?. You may then need to update the container whenever object changes. So i guess maintaining in-memory index for java objects is pretty difficult thing to do.

With such tools i think you can easily query moderate size java collections stored in Terracotta's durable memory with acceptable response time.

Tushar

Maven : Java Profiling

At workplace I use maven extensively. In fact I like maven so much that I am slowly moving lot of java eclipse project into maven project. Maven has very good eclipse plugin which makes integration with Eclipse very easy. You can run maven from eclipse. Maven is really good tool. If you have any specific needs you can write your own plugin. At my previous workplace, senior engineer in my project automated everything (starting jboss servers on remote machine in various environments : testing, perf-testing, integration testing etc. database population, database schema drop/rebuild etc).

But now I wanted to profile java application which I used to run from maven. One of the best part of maven is dependency management and repository - it builds classpath automatically for you, but then its pain too. If you want to run java application you have to do through Maven. There is exec plugin with which you can run any Application or shell script and there is exec:java with which you can run java Main class. The problem is that exec:java is in same JVM. So you cant run any java agent(-agent) or other things : Specifically Java Profiling. You should make sure you run the application within the same environment/settings.

So my first task was to get the complete classpath and then launch java with profiler java options. I am using jprofiler which uses JVMTI agent hence you need to append "-agentlib:jprofilerti=port=31757 -Xbootclasspath/a:/Applications/jprofiler5/bin/agent.jar" to java command line.

Here is little shell script through which I managed to do java profiling for maven Project. This will work only for J2SE applications tough!. Frustrating part was variable DYLD_LIBRARY_PATH. I was new to MacOS and was trying with usual LD_LIBRARY_PATH and -agentpath jvm option. Surprisingly -agentpath option should work on MacOS but it didnt work i guess some problem with Jprofiler binary. But lastly i managed to profile my application properly.


mvn dependency:build-classpath -Dmdep.outputFile=mycp.txt
export CLASSP=`cat ./mycp.txt`
export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/Applications/jprofiler5/bin/macos
$JAVA_HOME/bin/java -cp $CLASSP:./target/classes -agentlib:jprofilerti=port=31757 -Xbootclasspath/a:/Applications/jprofiler5/bin/agent.jar $*


This is the first time I had to do away with maven command. I wished somebody had written maven plugin for lauching jprofiler enabled apps. There is maven plugin for Yourkit Java Profiler : http://code.google.com/p/maven-yourkit-plugin/ but it did now work.

CountDownLatch for Terracotta

With Java 5, java has inbuilt concurrency library in java.concurrent with classes like CountDownLatch, CylcicBarrier, FutureTask, ExecutorService, LinkedBlockingQueue, ConcurrentHashMap, ReentrantReadWriteLock, which greatly simplifies writing multi-threaded applications. With increasing number of cores you need to write applications which are multi-threaded. With Terracotta this further makes really simple to run such application across more than one JVM effectively giving you more number of threads with slight degradation of performance but near-linear scalability. Terracotta supports some important data structures of "java.utl.concurrent" out of the box these are mainly : LinkedBlockingQueue, ExecutorService, CycliBarrier., FutureTask and of course Locks.


Below I am presenting one more addition to this library : CountDownLatch. CountDownLatch is used to co-ordinate between threads. You pass number of threads in constructor and each thread then calls countDown() method. When you want to get notified that all threads have finished their work you call await() method. This method will wait till all parties have finished and called countDown() method. If you want to write such code in Terracotta enabled applicaiton you have to use CyclicBarrier where each thread calls await() method. But this will cause finished threads to unnecessarily block on barrier. By using CountDownLatch you can "countdown" and exit the thread thus only master or co-ordinator thread needs to block.

Logic implemented is very simple - initiate with number of parties. decrease the counter in countDown method, when reached to zero notify all waiting threads and in await() method "wait()" on object till count is reached to zero.

Below is the source code for it. You need to put class MyCountDownLatch in instrumented classes section as well as define write-lock for the await() and countDown() method. You can download the source-code for the same here.


Main Class

public class MyCountdownLatch {

	

	

	int count = -1;

	public MyCountdownLatch(int count)

	{

		this.count = count;

	}

	

	

	public synchronized void countDown()

	{

		count--;

		if (count == 0) { notifyAll();  }

	}

	

	

	public synchronized void reset(int count)

	{

		this.count = count;

	}

	

	public synchronized void await() throws InterruptedException 

	{

        if (count == 0) { notifyAll(); return; }

        else { 

        	while(count > 0)

        		{

        			wait(); 

        		}

        }

	}



}

Test Class TestCountDownLatch

import java.util.Random;



public class TestCountDownLatch {

	

	public static int N=10;

	

	public static MyCountdownLatch startSignal = null;;

	 public static MyCountdownLatch doneSignal = null;	

	

	 private static Object lock = new Object();

	 

	public static void main(String[] args) {

		

		

		Runnable runs[] = new Runnable[N];

		

		if(startSignal==null)

		{

			synchronized (lock) {

				startSignal = new MyCountdownLatch(1);

				doneSignal = new MyCountdownLatch(N);	

			}

		}

		

		

		 for(int i=0;i		 {

			runs[i] = new Worker(startSignal,doneSignal);

			new Thread(runs[i]).start();

		 }

		 

		 startSignal.countDown();

		 try {

			doneSignal.await();

		} catch (InterruptedException e) {

			

			e.printStackTrace();

		} 

		

		

		System.out.println("All thread finished ...");

		

	}

	

	

	public static class Worker implements Runnable

	{

		MyCountdownLatch startSignal = null;

		MyCountdownLatch doneSignal = null;

	

		public Worker(MyCountdownLatch startSignal, MyCountdownLatch doneSignal)

		{

			this.startSignal = startSignal;

			this.doneSignal = doneSignal;

			

		}

		

		public void run()

		{

			System.out.println("Waiting for start signal...");

			try {

				startSignal.await();

			} catch (InterruptedException e) {

				

				e.printStackTrace();

			}

	        doWork();

	        doneSignal.countDown();

		}

		

		public void doWork()

		{

			System.out.println("Starting to work now");

			Random random = new Random();

			try {

				Thread.sleep(random.nextInt(2000));

			} catch (InterruptedException e) {

				// TODO Auto-generated catch block

				e.printStackTrace();

			}

			System.out.println("Completd work");

		}

		

	}

	



}

Config file tc-config.xml

<?xml version="1.0" encoding="UTF-8"?>

<tc:tc-config xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd" xmlns:tc="http://www.terracotta.org/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

  <servers>

    <server host="localhost" name="tc-srv01" bind="0.0.0.0">

      <data>%(user.home)/terracotta/server-data1</data>

      <logs>%(user.home)/terracotta/server-logs1</logs>

      <dso-port>9510</dso-port>

      <jmx-port>9520</jmx-port>

      <l2-group-port>9530</l2-group-port>

      <dso>

        <garbage-collection>

          <enabled>true</enabled>

          <interval>300</interval>

          <verbose>true</verbose>

        </garbage-collection>

      </dso>

    </server>

  </servers>

  <clients>

    <logs>%(user.home)/terracotta/client-logs1/</logs>

    <statistics>%(user.home)/terracotta/server-statistics-%D</statistics>

  </clients>

  <application>

    <dso>

    <instrumented-classes>

        <include>

          <class-expression>*..*</class-expression>

        </include>

      </instrumented-classes>

      <roots>

        <root>

          <field-name>TestCountDownLatch.startSignal</field-name>

        </root>

        <root>

          <field-name>TestCountDownLatch.doneSignal</field-name>

        </root>

      </roots>

      <locks>

        <autolock>

          <method-expression>* MyCountdownLatch.countDown(..)</method-expression>

          <lock-level>write</lock-level>

        </autolock>

        <autolock>

          <method-expression>* MyCountdownLatch.await(..)</method-expression>

          <lock-level>write</lock-level>

        </autolock>

      </locks>

    </dso>

  </application>

</tc:tc-config>

An Insane idea to Solve Java's Memory Problem

This could be insane idea to solve java's memory problem.

Garbage collection overhead has been one of the major performance problem. Recent improvements in Garbage collector : ability to parallel collect garbage using more processors or cores available, concurrent garbage collection has drastically reduced this belief that GC is mostly cause of performance problems. Earlier this problem could be solved by running multiple instances of application on the same box to use its all CPU power and memory. But for some memory-intensive application GC overhead is still a problem to be solved. As JVM is being touted as more of Virtual Machine for many interpreted languages as opposed to only Java runtime memory problem is bound to occur. I have been working recently on Terracotta which is Network Attached Memory (NAM) for Java Applications. When you say NAM, you application now can access more number of objects that could fit in your local heap since objects which are not in your local heap are transparently loaded into local JVM when they are accessed .. sort of lazy loading in hibernate. Now this idea can also be applied to JVM .. offload some objects which are not accessed often down to local disk and load them when they are not accessed.Imagine a situation where you are running application on 16 core or 32 core processors ( intel has one prototype of 80 cores) ..data processing ability of such hugh machine. With 64-bit platform JVM size can grow beyond 2 GB limit but when Full GC happens on JVM sized more than 4 GB its really painful for applications. I have never worked on such hugh JVMs no really no idea how jdk 5 and 6 performs in such situations. Sure there will be some optimizations in JVM to operate at such scale.

A simple prototype of this is various cache solutions which offer cache eviction to local disk when number of objects cross defined cache size. But these implementations are targeted as "caching solutions" which know only how to "get" and "put" objects in a manner so as to make best use of available space.

At first this idea may be insane but various optimizations can be done to make it practical. This is what terracotta has done, to avoid object serialization and operate at field level. Consider a map of 10000 fat objects. When this map is offloaded to disk all value objects will also be written to disk. Now when some value object is looked upon only the object can be loaded while rest of the objects still being stored on disk. This basically means you have to implement some sort of virtual memory manager for virtual machine which will use application access pattern and some stored intelligence to minimize the total load delay and at the same time allow application to operate on large data set.


Any comments are welcome .. I am sure there will be a lot of comments on this insane idea.

..
Tushar

Links : Kill Your Database with Terracotta

I am going to start a new series from this post. Many times you find great article on the internet that you would like to share with others. "Links" is one such series.


And very first link here is an Article written on Terracotta : An amazing technology for java application clustering. Its here : Kill Your Database. Terracotta is clustering technology with built-in support for HA. Clustering is all about shared data and when you talk about HA, you are actually caring for the shared data, its availability. Terracotta implements it by writing every change to shared data to disk in transactional manner. So if your data whose life is short or medium, (often you need to store derived data from the temporary data which is very large, and we use RDBMS for all of this) you can use terracotta to store it in very fast manner : since you don't have to convert data to relational model its all java. Pretty powerful!!. Other use case is Database off-loading which means you can use terracotta to store all data temporarily and then write to your database in manner which would not affect end-user response time. By doing this you remove database access and operation time from response time ( which mostly is the very significant part of the response time). Article mentioned above touches all these concepts. I will add one more link on terracotta org-site which describes this case ( and all other wonderful use-cases for terracotta). Here it is : Write Behind SOR. If you read my previous post, I had exactly discussed the same idea : Write Behind or Asynchronous writes. That time I knew only coherence supported this. Then I came to know that Gigaspaces also supports it. ( See comments below on the same post). But terracotta would be the most exciting among them all. Why? One reason is all existing application can easily integrate this pattern in there application with very small change in code-base. Second reason terracotta supports lot of java framework out of the box.

..
Tushar

JAMon Data for Java Applications

Recently i used JAMon for gathering important metric in one application which was not j2ee application. For JDK based application how do you get information in absence of JAMon Web Interface. I asked about it to Steve Souza who imeedaitedly returned with answer : MonitorFactory.getData() and MonitorFactory.getHeader(). Based on this data jsmons.jsp is written. For JDK based application i thought we can give same information via JMX. Below is small wrapeer to do

• Register JAmon Factory as MBean and implement some key functions like Attributes and Operations

   JAMonMBean mbean = new JAMon();
   ObjectName name = new ObjectName("jamon.perf:type=JAMonBean");
   MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
   mbs.registerMBean(mbean, name);

• JAMon Client

   ObjectName name = new ObjectName("jamon.perf:type=JAMonBean");
   MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
   String arr[] = {key};
   String[] signature = new String[] { "java.lang.String" };
   obj = mbs.invoke(name,operationName, arr,signature );

• Start background thread to store information in file

   Object[][] data= client.getData();   
   // record format date : label : Hits : Avg : Total : StdDev
   String timestamp = "" + System.currentTimeMillis();
   if(data!=null)
   {
    for(int i=0;i<data.length;i++)
    {
     StringBuilder sb = new StringBuilder();
     sb.append(timestamp).append(" : ");
     sb.append(data[i][0]).append(" : ");
     sb.append(data[i][1]).append(" : ");
     sb.append(data[i][2]).append(" : ");
     sb.append(data[i][3]).append(" : ");
     sb.append(data[i][4]);
     pw.println(sb.toString());
     pw.flush();
     //System.out.println(sb.toString());
    }
   }

Equipped with above, I have also written a small utility which outputs information of all counters periodically in the format

timestamp : label : avg. response time : hits : std. deviation

I have uploaded code used to demonstrate the JAMon Bean have been uploaded at http://tushar.khairnar.googlepages.com/perf-sample.zip.

Please see Sample Program for the same.

Java Performance : Monitoring and Measurements - APM

Recently I came across this (Run-time performance and availability monitoring for Java systems) wonderful series of articles about implementing run-time performance monitoring for application's ecosystem sometimes its called APM (Application Performance Management). This series really good piece of information every person who needs to implement some kind of ecosystem for performance needs to read it. But all screen shots resemble Wily Introscope so basically it describes everything what Introscope does.
Article say Helios is the reference implementation of idea discussed.

It covers

• Monitoring fundamentals - Why you need it. What you need it - Periodic reports based on template and custom reports, Historical storage and analysis, Live Visualization and simultaneous plotting for correlation, Alerting : based on email, blackberry, sms or JMS with GUI, Dashboards


• Recent advancements - like Agentless monitoring, Synthetic Transaction


• Some very high-level design details - Performance Data Source, Collector, Tracers
  


• Tracing patterns : Polling, Listening, Interception , Instrumentation

And about how you go about performance monitoring in general?

It focuses JMX as primary way of doing things but if you look towards commercial APM like CA Wily's Introscope these are not JMX based. I feel if your application is java centric then it helps to have java based apm : Introscope Wily.

I would like to add following too.

Synthetic transactions : I feel synthetic transactions are really helpful but at the same time difficult to implement. Tools like Grinder can help you to implemented these.

Application Specific Metrics : Generally you have infrastructure metrics like os metrics, storage metrics, network and application infrastructure metrics like : App Server metrics (queue length, response time) and Database : avg. query time. But sometime you may require to gather application/business specific metrics. An example would be say : policies issued per day. Framework should be flexible to allow such metric to be posted to APM where it can then be correlated to technical metrics.

Dignosis : Also APM should be able to switch gears when problem occurs. When problem occurs APM should start collecting data at more granular level so as to get more refiend picture of system. Introscope has dignosis tool called : Transaction Monitor which traces entire transaction as one context. Article though talks about "heavy instrumentation" as i guess there

Heuristics: APM should also provide some level of analysis (rule based) like what Glassbox does. See demo http://demo.glassbox.com/glassbox

Dilemma with APM systems is that commercial products like IBM Tivoli, HP OpenView, Mercury BAC come with many features with hefty cost. So open-source comes to rescue here : There lot of Open-source projects which implement this idea. I am big fan of Nagios. Others like Groundwork Monitor implement extra functionality around nagios. Nagios 3.0 has made lot of progress and now "installable" for normal user with this guide. http://nagios.sourceforge.net/docs/3_0/quickstart.html
Also, if your application is java centric then it makes good seance to have JMX based system as described in this article. Sometimes you don't need full blown system. In such situation tools like JAMon API can help you. Do visit JAMon site it will surely help you even its recommended to keep running on production environment.

If you are interested in java profiling and how its done read : Build your own profiling tool and Jensor (jensor.sourceforge.net). Jensor is java profiler built by TCS's Performance Engg Group ( from where i started my career) is focused on first article mentioned but has good analysis gui which helps you to dig in. Commercial profilers JProbe and JProfiler are really good.

In past I had experimented same idea with system called Nagios. There focus was to implement performance monitoring system for the entire lab. We had written Nagios plugins for Webspere(PMI based) and weblogic ( weblogic shell) for collecting performance metrics. Nagios but, is only scheduling and executing engine which does not effectively care about the data ( Currently with 3.0 version it has lot of extension points where you can save events and performance data in mysql or postgres database). So we had used small open-source system Perfparse for performance data and pulling out reports. In our lab we had implemented this on about 80 servers on single CPU monitoring machine running RHEL 4.0. Worked very well for me.

Best one is the one which solved your problem!!.

..
Tushar

Java Performance - Multi Core Processors

Hi i was reading book called Java Concurrency in Practice by Brian Goetz and others.
I found this interesting quote :


"For the past 30 years, computer performance has been driven by Moore's Law; from now on, it will be driven by Amdahl's Law. Writing code that effectively exploits multiple processors can be very challenging."

Amdahl's law describes how much a program can theoretically be sped up by additional computing resources, based on the proportion of parallelizable and serial components.

util.concurrent from Doug Lea brought Java Concurrency API right into JDK. From starting java was the first (at first among mainstram prograaming languages)
language to support Multithreading. With JAVA 5 any developer can write safe and scalable Java programs.


All latest processors now are multi-core processors this clearly shows shift towards parallel systems where your program has to effectively use all hardware thread underlying cpu provides. Unless programs are completely multithreaded, they simply won’t use the power available in hugely multicore systems. Lot of attention is given to transform code to multi-thread code. Read this post for details : "Intel: We Can Transform Single Thread to Multithread"

I also got to read this wonderful post which discusses Java Performance mainly GC on multicore processors : Multicore may be bad for Java.

Indeed Java is Multicore Ready I believe .. Happy Multithreading.

..
Tushar

Java Performance : Caching Clustering ... and "FlushCache"

Hi i came back after long time. This time again with java. Recenly playing with lot of java solutions relating to scaling and clustering. Lot of places java high performacne systems you see buzzwords like multi-threading, clustering, caching, map-reduce, partitioning, grid solutions. Lets discuss some of them.

I have been using hibernate for last 1.5 year in my personal experiments and at work place too. In one of project we had cache requirement for master data which was read-only. There we had implemented caching by hand as follows.

1. Wrapper around DAOs which first check in cache and then go to database.
2. Cache was implemented with Websphere provided object pooling API.
3. Websphere provided object pooling mechanism which even works in clustered set-up or network deployment with cache replication strategies.

Our requirement was well met and caching brought lot of improvement in response time as expected.

After caching, horizontal scaling or clustering comes into my mind when i think about performance.
Since then, I discovered lot of caching/clustering projects/apis, some of which are very much i feel worth to take a note.

1. Tangosol - recently acquired by Oracle, along with oracle, timesten ( oracle in-memory database, another acquisition :-) ) forms formidable data-tier. Tangasol is very proven prodcut which provides lot of features like distributed cache, data partitioning. Cameron Purdy on Theserverside claims to reach upto 0.5 milllion cache transaction per second.

2. Terracotta :- Terracotta is very revolutionary java product in fact listed in top 10 java things of year. Terracotta actually speaking is not caching solution but a clustering solution. Its basically JVM level clustering. They have provided cache solutions for lot of commons requirements : Hibernate, HTTP Session, Spring Beans etc. They provide good extension for lot of open-source projects. I had tested tomcat clustering and terracotta, similar to one on terracotta site and it gives good performance boost. Among all other clustering solutions terracotta has simplest programming model : NO PROGRAMMING MODEL. right : objects basically clustered at jvm level so only configuration change no code change. Practically you do require to change java code but that's minimal All java semantics work well in terracotta cluster. Terracotta is very useful in patterns like Master/Worker or bunch processes looking for some coordination or data sharing. Master Worker pattern term i guess was introduced much before Googles Map/Reduce and very much the same idea.


3. Memcached : this one is in c but has client libraries in almost all major programming languages. Idea of memcached is actually little bit different. You keep bunch of memcached process running objects are stored on these processes with hash key. Memcached works best when its processes run on web servers which are cpu-intensive with lot of memory available.


4. New bunch of grind frameworks : gigaspaces, hadoop - java map-reduce implementation

But now i had come across very different requirement where cache modification (writes) were significant in numbers and jdbc-batching with asynchronous operation was key to performance sometime this is called - write behind. When i looked upon only tangosol claims to have such facility so i decided to implement by hand. Here is an idea.

1. A background threads basically monitors queue.
2. All cache write operations append modified objects to this queue.
3. queue periodically flushes them to database and notifies successful operation to clients.


Here catch is how do you handle object graph updates
1. Delta Calculation
2. New Object insertion may require updates in one specific order where result of first inserts basically required for next one. (foreign key)

Initially i went with hibernate where i used StatelessSession as SQL generation engine hoping that hibernate will calculate delta properly. But lack of L1 cache means no life cycle hence no delta monitoring. So i had two choices.

1. Use merge : Fires extra selects
2. Generate sql by hand.

With help of cglib proxies i managed to get list of dirty columns but then how do you handle object relationships?

I soon realized whole ORM needs to be implemented... here list of requirements for basic ORM cum cache with write-behind

1. easy configuration : declarative orm mapping sufficient for most of scenarios
2. Should provide different execution strategies timer driven, threadpool driven, batch reached, batch and timer combined, entity level strategy for sync
3. in-memory multi-indexing based on columns or own implementation : cache object can be queried with different criteria
4. notification - listeners
5. target rate with % of write operation : 500 requests/second.
5. recovery test : check points : automated and manual
6. clustering with help of terracotta.

This is exhaustive list .. where first one itself is very big one..... my take on it is to restrict the scope and focus on caching instead of fancy orm stuff the one like hibernate.... it should be able to support only object graph ... all lifecycle is left to user..

I will add code snippets in coming posts ... now only idea has been finalized.

How to integrate DWR and Struts

DWR is AJAX framework for JAVA. Using DWR you can call directly any method of java class asynchronously ie. without entire page reload. With following you can even integrate DWR with struts so as to have access to important formbeans and also any other java classes. Only restriction being : methods to be called has to be form methods. This is easily done by refactoring code in Action class to form class. By this way you can call this code from Action class as well as via ajax.

Add dwr.jar to web project

Modify Web.xml Add following code to web.xml. Make sure DWR servlet gets loaded after ActionServlet.

<servlet>
    <servlet-name>dwr-invoker</servlet-name>
    <servlet-class>org.directwebremoting.servlet.DwrServlet</servlet-class>
    <init-param>
      <param-name>debug</param-name>
      <param-value>true</param-value>
    </init-param>
    <init-param>
      <param-name>activeReverseAjaxEnabled</param-name>
      <param-value>true</param-value>
    </init-param>
    <init-param>
      <param-name>initApplicationScopeCreatorsAtStartup</param-name>
      <param-value>true</param-value>
    </init-param>
    <init-param>
      <param-name>maxWaitAfterWrite</param-name>
      <param-value>100</param-value>
    </init-param>
    <!--
    <init-param>
      <param-name>org.directwebremoting.extend.ServerLoadMonitor</param-name>
      <param-value>org.directwebremoting.impl.PollingServerLoadMonitor</param-value>
    </init-param>
    -->
    <load-on-startup>2</load-on-startup>
  </servlet>

  <servlet-mapping>
    <servlet-name>dwr-invoker</servlet-name>
    <url-pattern>/dwr/*</url-pattern>
  </servlet-mapping>

Create dwr.xml file. This file will tell DWR which classes are to be exposed for asynchronous calls. Following example file exposes PersonFrom class using interface name Forms using Struts Creator. It also exposes Date class as remote class JDate.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE dwr PUBLIC "-//GetAhead Limited//DTD Direct Web Remoting 2.0//EN" "http://getahead.org/dwr//dwr20.dtd">
<dwr>
 <allow>
  <create creator="new" javascript="JDate">
   <param name="class" value="java.util.Date" />
  </create>
  <create creator="struts" javascript="forms">
   <param name="formBean" value="personForm" />
  </create>
  <convert converter="bean" match="$Proxy*" />
 </allow>
</dwr>

Include following lines in JSP for adding DWR javascripts. “forms.js” refers to our “forms” class exposed in dwr.xml. These files do not exist physically but are dynamically generated and served by DWR servlet.

<script type='text/javascript' src='dwr/engine.js'> </script>
<script type='text/javascript' src='dwr/util.js'> </script>
<script type='text/javascript' src='dwr/interface/forms.js'> </script>

Now you can call any method of form PersonForm class. PersonForm class has method called generateAntiSpamMailto. You can directly call this method using javascript. See following example. Following example basically dynamically calls “generateAntiSpamMailto” method on server to get anti-spam text and puts in to the div and makes the div visible. This is done by defining function as the third argument. First two arguments are regular arguments which are same as java arguments.

function process() {
  var address = dwr.util.getValue("address");
  var name = dwr.util.getValue("name");
  alert('Addres is ' + address);
  alert('Name is ' + name);
  
  forms.generateAntiSpamMailto(name, address, function(contents) {
    alert('content is ' + contents);
    dwr.util.setValue("outputFull", contents, { escapeHtml:false });
    dwr.util.byId("output").style.display = "block";
  });
}

and HTML part as

<input id="submit" type="button" value="Submit Query" onclick="process()"/>
<html:submit>Submit Query</html:submit>
</html:form>

<div id="output" style="display:none;">
  <h2>Generated Links</h2>
  <textarea id="outputFull" rows="9" cols="70">
  </textarea>
</div>

You can even use DWR to validate certain field on the spot when user finishes typing the data. Validation will be done on the server but called asynchronously via AJAX. Following is an example which makes use of Apache Commons Validator framework to validate an email address. This require certain jars to added to lib(BSF, BSH,Commons jars included in the zip file attached).

DWR xml

<create creator="script" javascript="EmailValidator" scope="application">
<param name="language" value="beanshell"/>
<param name="script">
import org.apache.commons.validator.EmailValidator;
return EmailValidator.getInstance();
</param>
</create>

and HTML /JavaScript part

<script> 
function verifyAddress() {
  var address = dwr.util.getValue("address");
  EmailValidator.isValid(address, function(valid) {
    dwr.util.setValue("addressError", valid ? "" : "Please enter a valid email address");
  });
}
</script>

<html:text property="name" size="16" onkeypress="dwr.util.onReturn(event, process)" onblur="verifyAddress()"/>