What is lucene-bytebuffer.
lucene-bytebuffer is Lucene Directory implementation using Direct ByteBuffer. Directory in lucene is backing storage for index. Lucene uses directory for storing index contents. So there is RAMDirectory, FileDirectory, MemoryMappedFileDirectory, NIODirectory each presenting various different options. lucene-bytebuffer will allow in-memory index to grow upto several gigabytes without incurring garbage collection cost.
Mostly indexes are 90 to 95% read and 2-5% write ie. index hardly changes. If index is huge it will cost a lot in terms Garbage Collection CPU cycles. RAMDirectory holds arrays of size 1024 so for 1GB index its 1 million array objects. So as size gets increased in-memory index performance degrades due to garbage collection.
What if you want to index say 5GB data? Use off-heap bytebuffer backed directory.
Another question is why would you want to use lucene in-memory indexing. May be as Cache which can be queried on more than one property of object indexed?
Tuesday, October 19, 2010
jmalloc : Manual Memory Management in java
One of the definitive advantage of Java over C++ is automatic memory management. Automatic memory management also has its cost tough. This cost was high in earlier versions of JDK(Serial stop the world garbage collection) but now it has been much improved : parallel and concurrent collections. In some cases this cost is so high that manual memory management is the answer. jmalloc is one such simple effort.
GC pain point in java, a limiting factor in many cases. Garbage collection tuning in java is considered as black art and very difficult to tune.
Caching provides performance boost for lot of application. Caching of large data is restrictive because caching mostly is very small part of application logic but it costs relatively more in terms GC impact. JVM don't perform well predictively beyond size of 4GB. Cache is typical - it holds objects with predictable life cycle. Some objects infact live through-out the application life such as "reference data" which does not change and remain cached. Such objects are also problematic for GC, they get promoted to old generation and scanned in every Full garbage collection wasting CPU. Terracotta has addressed similar problem using direct ByteBuffer. jmalloc also does the same thing for ehcache.
BigMemory benchmark claims to have been scaled upto 350 GB of cache on beefy server. BigMemory has shown that garbage collection that java offers is not sufficient for some use-cases like Caching. Caching modifies object only two times : Put on Cache and Eviction from Cache. This is typical case of manual memory management. jmalloc is manual memory management of direct buffers with two simple routines : malloc and free. Direct buffers are not visible to java garbage collection. Thus object stored in directbyte buffer lives as long as its ByteBuffer reference is not collected. jmalloc allocates a single ByteBuffer and divides it into many variable size chunks where objects are serialized and stored.
This is just start. Apart from generic malloc/free metods, I am planning to write a helper class for ehcache which will wrap ehcache so that all benefits like eviction, disk based overflow are available but the object is stored in
If you like the idea let me know.
..
Tushar
GC pain point in java, a limiting factor in many cases. Garbage collection tuning in java is considered as black art and very difficult to tune.
Caching provides performance boost for lot of application. Caching of large data is restrictive because caching mostly is very small part of application logic but it costs relatively more in terms GC impact. JVM don't perform well predictively beyond size of 4GB. Cache is typical - it holds objects with predictable life cycle. Some objects infact live through-out the application life such as "reference data" which does not change and remain cached. Such objects are also problematic for GC, they get promoted to old generation and scanned in every Full garbage collection wasting CPU. Terracotta has addressed similar problem using direct ByteBuffer. jmalloc also does the same thing for ehcache.
BigMemory benchmark claims to have been scaled upto 350 GB of cache on beefy server. BigMemory has shown that garbage collection that java offers is not sufficient for some use-cases like Caching. Caching modifies object only two times : Put on Cache and Eviction from Cache. This is typical case of manual memory management. jmalloc is manual memory management of direct buffers with two simple routines : malloc and free. Direct buffers are not visible to java garbage collection. Thus object stored in directbyte buffer lives as long as its ByteBuffer reference is not collected. jmalloc allocates a single ByteBuffer and divides it into many variable size chunks where objects are serialized and stored.
This is just start. Apart from generic malloc/free metods, I am planning to write a helper class for ehcache which will wrap ehcache so that all benefits like eviction, disk based overflow are available but the object is stored in
If you like the idea let me know.
..
Tushar
Thursday, September 16, 2010
BigMemory - Memory with no garbage collection overhead How?
I just came to know about BigMemory. After reading description and Terracotta CTO Ari's blog I wondered how does it work? BigMemory stores object in memoey without any garbage collection tax and stores it in native memory. It means you are no longer bound by JVM heap limit and you can use all memory (which is dirt cheap) available. And all this in Plain Java. No JNI.
I mean how did it happened that If there was technique available why cant any available cache framework used it? By implementing the this technique along with complete memory manager, Terracotta thus showed that its indeed leader in distributed cache. Why spent effort in optimizing garbage collection times which is a black art.
Lets assume that We have such mechanism which allows us to store java objects in native memory. So how do we implement BigMemory like system. Let's focus just on Cache usecase . You need to track objects in cache. But Object tracking is quite easy in cache usecase. They are removed directly by cache eviction threads or user so ultimately its map.remove(key).
So how does BigMemory work?. This is just my guess. It may be using Direct ByteBuffers. I did some googling around allocating native memory in JVM and found this article. and Yes Direct ByteBuffers are stored in Native memory. So below is my little shot at mimicking BigMemory. Basically each object is converted in ByteBuffer and stored in ByteBuffer. Some time and memory can further be saved by using faster and more smart serialization techniques.
The solution above stores keys in JVM heap and objects on non-heap memory. So If number of objects are 2M, JVM only has overhead of 2M objects while doing GC and not 4M ( in case of Map and 4-5M in case of EHCACHE where object is serialized and wrapped in ehcache Element object). This can be even further reduced by implementing some sort of hashing. Key is hashed into one of the buckets which are serialized and stored in buffers. You de-serialise bucket metadata and then iterate though it to find out actual buffer and then locate element. But BigMemory is just not cache its entire object graph storage.
Thus version presented here is poor man's BigMemory compared to what BigMemory will have but it still can save hugh GC tuning efforts in cases where there is hugh memory requirement in the application and thus heavy garbage collection. But Terracotta's BigMemory would be with much more smart strategies and data structures, integrated with distributed garbage collector and implementing all cluster semantics as all other terracotta clustered data structure do. Waiting for integration of BigMemory into Terracotta server. Memory has been one of the biggest paint points in Terracotta. That would give Terracotta unlimited-scalability in terms of number of distributed objects stored.
I mean how did it happened that If there was technique available why cant any available cache framework used it? By implementing the this technique along with complete memory manager, Terracotta thus showed that its indeed leader in distributed cache. Why spent effort in optimizing garbage collection times which is a black art.
Lets assume that We have such mechanism which allows us to store java objects in native memory. So how do we implement BigMemory like system. Let's focus just on Cache usecase . You need to track objects in cache. But Object tracking is quite easy in cache usecase. They are removed directly by cache eviction threads or user so ultimately its map.remove(key).
So how does BigMemory work?. This is just my guess. It may be using Direct ByteBuffers. I did some googling around allocating native memory in JVM and found this article. and Yes Direct ByteBuffers are stored in Native memory. So below is my little shot at mimicking BigMemory. Basically each object is converted in ByteBuffer and stored in ByteBuffer. Some time and memory can further be saved by using faster and more smart serialization techniques.
public Object put(K key, V value){
try {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream stream = new ObjectOutputStream(baos);
stream.writeObject(value);
stream.close();
byte b[] = baos.toByteArray();
ByteBuffer buffer = ByteBuffer.allocateDirect(b.length);
ByteBuffer buffer2 = map.put(key, buffer);
if(buffer2!=null){
ByteArrayInputStream bais = new ByteArrayInputStream(buffer2.array());
ObjectInputStream oois = new ObjectInputStream(bais);
V object = (V)oois.readObject();
oois.close();
return object;
}
else return null;
} catch (IOException e) {
e.printStackTrace();
throw new RuntimeException(e);
} catch (ClassNotFoundException e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
public V get(Object key){
try{
ByteBuffer buffer2 = map.get(key);
if(buffer2!=null){
ByteArrayInputStream bais = new ByteArrayInputStream(buffer2.array());
ObjectInputStream oois = new ObjectInputStream(bais);
V object = (V)oois.readObject();
oois.close();
return object;
}
else return null;
}catch (IOException e) {
e.printStackTrace();
throw new RuntimeException(e);
} catch (ClassNotFoundException e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
The solution above stores keys in JVM heap and objects on non-heap memory. So If number of objects are 2M, JVM only has overhead of 2M objects while doing GC and not 4M ( in case of Map and 4-5M in case of EHCACHE where object is serialized and wrapped in ehcache Element object). This can be even further reduced by implementing some sort of hashing. Key is hashed into one of the buckets which are serialized and stored in buffers. You de-serialise bucket metadata and then iterate though it to find out actual buffer and then locate element. But BigMemory is just not cache its entire object graph storage.
Thus version presented here is poor man's BigMemory compared to what BigMemory will have but it still can save hugh GC tuning efforts in cases where there is hugh memory requirement in the application and thus heavy garbage collection. But Terracotta's BigMemory would be with much more smart strategies and data structures, integrated with distributed garbage collector and implementing all cluster semantics as all other terracotta clustered data structure do. Waiting for integration of BigMemory into Terracotta server. Memory has been one of the biggest paint points in Terracotta. That would give Terracotta unlimited-scalability in terms of number of distributed objects stored.
Thursday, April 1, 2010
More than MVC framework for Flex : Mate
Recently at work place I was working on complex GUI screen which was to be designed in Flex. Flex has quite nice capabilities : Making beautiful GUI containing charts, widgets with minimal efforts and getting data from java-based-server-side with RemoteObject. But I guess as application will get complex(Do check grooveshark.com to see how rich and complex but beautiful GUI can be created with Flex) I needed some way of structuring and designing application so anybody can understand and extend application without reading thousands of lines of action script code. Being java programmer (framework based eco-system) naturally I thought on the lines of Struts, MVC framework. But then I felt need of spring like container too where I can write/design each component separately, test them and inject and wire them in main application. There are lots of framework in flex : Cairngorm, pureMVC and Mate. Naturally first pick was Cairngorm since its official adobe framework. While browsing through HelloWorld program, I thought why I am needed to to make so many framework specific classes. Last update date on download site was very old : 30th May 2007. Then I learned about Mate : a wonderful framework which allows structuring of application in very unobtrusive way with very small learning curve. It was quite easy for me to help other team members understand new design with this framework.
Mate uses flex eventing. So with Model-View-Controller, View generates events like Submit-Form which then you can "map" to one or more Controllers. The best feature of this framework is "Injection". It injects model into controller and view so once you change model in Controller methods View updates it self. With Flex data binding and Mate injection this becomes quite powerful and very simple design for complex GUI. Mate EventMap tag allows declarative coding thus allows rapid testing and prototyping. When you look at Mate EventMap you will come to know what Application is doing, its high-level components. - your design - exactly what framework should do. Mate allowed me structure my application and set discipline.
Say for example on clicking on "submit" button, you want to show make remote call, get data, update couple of labels on screen and put data into Grid also calculate Total Amount from all records and based on some "Value" you want to show some Signal like Yellow, Amber or Green indicating current status. Such screen can be described by simple EventMap declaration as follows. View can be implemented with its own Panels : SearchPanel and DisplayPanel. SearchController will host all activity specific code like calculating Totals or Summary etc.
EventMap
View code can be refactored into two panels : SearchPanel and DisplayPanel SearchPanel looks like as follows :
SearchPanel
Where as DisplayPanel looks like as follows :
DisplayPanel
and code below injects model into controller and DisplayPanel. The way it works like Spring context's singlton bean so same instance is shared and below tag says : DisplayPanel.searchResult = SearchController.currentResultSet
EventMap - Injector
Below is code for Controller which requires its own Binding when currentResultSet changes :
SearchController
One more good feature if Mate is that it has MockService interface where you can write dummy remote service as part of Flex application itself. This allows developers to work independently when server-side services are not ready or avaialable.
Welcome to awesomeness of Mate.
Tushar
Mate uses flex eventing. So with Model-View-Controller, View generates events like Submit-Form which then you can "map" to one or more Controllers. The best feature of this framework is "Injection". It injects model into controller and view so once you change model in Controller methods View updates it self. With Flex data binding and Mate injection this becomes quite powerful and very simple design for complex GUI. Mate EventMap tag allows declarative coding thus allows rapid testing and prototyping. When you look at Mate EventMap you will come to know what Application is doing, its high-level components. - your design - exactly what framework should do. Mate allowed me structure my application and set discipline.
Say for example on clicking on "submit" button, you want to show make remote call, get data, update couple of labels on screen and put data into Grid also calculate Total Amount from all records and based on some "Value" you want to show some Signal like Yellow, Amber or Green indicating current status. Such screen can be described by simple EventMap declaration as follows. View can be implemented with its own Panels : SearchPanel and DisplayPanel. SearchController will host all activity specific code like calculating Totals or Summary etc.
EventMap
- <EventHandlers type="{SearchEvent.SEARCH}">
- <!-- call the remoting service -->
- <RemoteObjectInvoker instance="{services.productService}" method="search" arguments="{event.searchCriteria1,event.searchCriteria1}">
- <!-- result sequence gets executed when service returns with a result -->
- <resultHandlers>
- <MethodInvoker generator="{SearchController}" method="setGridData" arguments="{resultObject}"/>
- <MethodInvoker generator="{SummaryCalculator}" method="calculateSummary" arguments="{resultObject}"/>
- <MethodInvoker generator="{SignalIndicator}" method="calculateAndIndicateState" arguments="{resultObject}"/>
- </resultHandlers>
- <faultHandlers>
- <CallBack method="handleFault" arguments="{fault.faultDetail}"/>
- </faultHandlers>
- </RemoteObjectInvoker>
- </EventHandlers>
View code can be refactored into two panels : SearchPanel and DisplayPanel SearchPanel looks like as follows :
SearchPanel
- <mx:Script>
- <![CDATA[
- import mate.events.*;
- import mx.controls.Alert;
- import mx.collections.*;
- private function fireSearch():void
- {
- var event:MessageEvent = new SearchEvent(SearchEvent.SEARCH, true);
- event.searchCriteria1 = textInput1.text;
- event.searchCriteria2 = textInput2.text;
- dispatchEvent(event);
- }
- ]]>
- </mx:Script>
- <mx:Button id="searchBtn" label="Search" width="100" click="fireSearch()"/>
Where as DisplayPanel looks like as follows :
DisplayPanel
- <mx:Panel>
- <mx:Script>
- <![CDATA[
- import mate.events.*;
- import mx.controls.Alert;
- import mx.collections.*;
- import mate.model.*;
- [Bindable]
- public var searchResult:SearchResultVO;
- ]]>
- </mx:Script>
- <mx:VBox label="Search Result">
- <mx:DataGrid id="dataGrid" width="350" height="200" dataProvider="{searchResult.Data}"/>
- </mx:VBox>
- </mx:Panel>
and code below injects model into controller and DisplayPanel. The way it works like Spring context's singlton bean so same instance is shared and below tag says : DisplayPanel.searchResult = SearchController.currentResultSet
EventMap - Injector
- <Injectors target="{DisplayPanel}" >
- <PropertyInjector targetKey="searchResult" source="{SearchController}" sourceKey="currentResultSet" />
- </Injectors>
Below is code for Controller which requires its own Binding when currentResultSet changes :
SearchController
- package mate.controller
- {
- import flash.events.Event;
- import flash.events.EventDispatcher;
- import mx.controls.Alert;
- import mx.collections.ArrayCollection;
- import mate.model.*;
- public class SearchController extends EventDispatcher
- {
- private var _currentResultSet:SearchResultVO;
- [Bindable (event="currentSetChange")]
- public function get currentResultSet():SearchResultVO
- {
- return _currentResultSet;
- }
- public function setGridData(result:SearchResultVO):void
- {
- // Do other processing if required....
- _currentSet = result;
- dispatchEvent( new Event('currentSetChange'))
- }
- }
- }
One more good feature if Mate is that it has MockService interface where you can write dummy remote service as part of Flex application itself. This allows developers to work independently when server-side services are not ready or avaialable.
- <MockRemoteObject id="helloService" showBusyCursor="true" delay="1" mockGenerator="{MockHelloService}"/>
Welcome to awesomeness of Mate.
Tushar
Sunday, March 14, 2010
Uploading and Sharing Photos
We all want to share pictures all our Kodak moments with our friends. This is one kind of existence in Web 2.0 virtual world. This normally involves following pattern :
Picasa makes sharing photos really easy. I use Picasa Desktop Application for all my photos. Picasa has concept of gadgets where you can add button, clicking on which it will post your album to sites like Orkut or Facebook. So here is my way.
And orkut has inbuilt support for Picasa Web Albums See screen shot below
Hope this simple tip makes your life with Photos and Social Networks easier.
Tushar.
- Take a lot of pictures by using point-and-shoot digital cameras.
- Download them from Camera/Memory card to computer
- Edit them in Picture editor
- Upload them to web-album sites like flickr or picasa
- Share photos from Picasa or Filkr
- Upload and share on social networking sites like Orkut or Facebook.
Picasa makes sharing photos really easy. I use Picasa Desktop Application for all my photos. Picasa has concept of gadgets where you can add button, clicking on which it will post your album to sites like Orkut or Facebook. So here is my way.
- Download photos from Camera or Phone using Picasa.
- Edit them using picasa
- Use picasa tool bar buttons to upload them to Picasa Web album, orkut and facebook.
And orkut has inbuilt support for Picasa Web Albums See screen shot below
Hope this simple tip makes your life with Photos and Social Networks easier.
Tushar.
Wednesday, February 10, 2010
Google Buzz
My first impression - I like the idea of adding connected sites : twitter, picasa, blogger, reader and all status messages. I am still searching for good social network desktop client which will connect to all my accounts : gmail, orkut, facebook, yahoo, linked-in, twitter but there is not such application existing yet. The once which was closer is Digsby, but digsby's UI is really clumsy to use.
Buzz is single window solution. You want to share picasa album or your recent tweet it works. Lets hope that it does not fail as Google Wave failed due to absence of freinds. I hope google this time sends invitations to clusters of users such that when I use buzz I will find at least some friends to share buzz. Buzz is suppose to compete with Facebook and Twitter, it surely has plus points against both
On other side, I don't know what google is thinking of having two different products : Wave and Buzz. I think since Wave has failed Google was in serious need to restrict facebook's advance so google added social networking aspects to its most loyal user base - Gmail. I am not sure about Buzz success, its surely going to tank as like google's other recent products. Facebooks strong point is that all sharing links, photos, activities, thoughts is at the central of page but in gmail is just another view so am not sure it should be part of gmail. Gmail view is OK but there should be separate page also where I can just spend time on Buzz. Hope google adds new features slowly.
Buzz is single window solution. You want to share picasa album or your recent tweet it works. Lets hope that it does not fail as Google Wave failed due to absence of freinds. I hope google this time sends invitations to clusters of users such that when I use buzz I will find at least some friends to share buzz. Buzz is suppose to compete with Facebook and Twitter, it surely has plus points against both
- it works right inside the most popular Google application - gmail
- no 140 character limit
- auto publish feature : when I tweet, same twit will be "buzzed" too
On other side, I don't know what google is thinking of having two different products : Wave and Buzz. I think since Wave has failed Google was in serious need to restrict facebook's advance so google added social networking aspects to its most loyal user base - Gmail. I am not sure about Buzz success, its surely going to tank as like google's other recent products. Facebooks strong point is that all sharing links, photos, activities, thoughts is at the central of page but in gmail is just another view so am not sure it should be part of gmail. Gmail view is OK but there should be separate page also where I can just spend time on Buzz. Hope google adds new features slowly.
Saturday, January 9, 2010
Google Nexus One
Two days back I bought my android phone in US. Cant wait to get hands on it. Here are few pics of it.
The most astonishing thing is that it got 1 GHz CPU while most of the smart phones in current generation are around 528 MHz thats big advantage and its got bigger screen. For screen bigger the better. It also has Flash Player, significant factor over IPHONE 3GS.
Tech Specifications. For details spec see : Google Site
The most astonishing thing is that it got 1 GHz CPU while most of the smart phones in current generation are around 528 MHz thats big advantage and its got bigger screen. For screen bigger the better. It also has Flash Player, significant factor over IPHONE 3GS.
Tech Specifications. For details spec see : Google Site
- Processor Qualcomm QSD 8250 1 GHz
- Android Mobile Technology Platform 2.1 (Eclair)
- 512MB Flash 512MB RAM 4GB Micro SD Card (Expandable to 32 GB)
- Display 3.7-inch (diagonal) widescreen WVGA AMOLED touchscreen 800 x 480 pixels
- Camera 5 megapixels Autofocus LED flash with Video captured at 720x480 pixels at 20 fps
- 3G : UMTS Band 1/4/8 (2100/AWS/900)
- Wi-Fi (802.11b/g)
- Bluetooth 2.1 + EDR
Saturday, August 22, 2009
Querying Java Objects stored in Terracotta's NAM Part 3
First and Second part of this series I talked about Querying data structures available in Java. First part specifically talked about existing ones like JoSQL, JxPath and Quaere and discussed indexing problem. Second part specifically talked about lucene and jofti as indexing frameworks and wrote small framework to test and get some performance numbers. This test showed that B-tree index (memory and possibly disk based) is suitable and not lucene index. Third part of this series now I am discussing my own small framework- tcquerymap which is rewrite of Jofti. Jofti is old and not maintained and uses jdk14 libraries. Now even documentation link does not work : http://prism-index.com/guide.html.
querymap documentation link : Getting Started With Querymap
querymap source download : Complete Source Code with Eclipse Project
querymap TIM : Copy this TIM to terracotta modules directory to use it
querymap terracotta sample app : Download sample eclipse App. You can use Terracotta eclipse plugin to launch it within eclipse
So the basic idea is maintaining in-memory indexes which index String Ids with "Comparable" as Keys. All primitive wrappers in Java implement this interface so no special conversion is needed.
How it works
It scans java objects and makes comparable objects for each of the property mentioned to be indexed. It inserts these comparable objects in its own tree against String Ids assigned to the java object. So in-effect its nothing more than maintaining many in-memory Maps. In fact the implementation that I wrote uses JDK TreeMap and not b-tree map. But in-future it will be replaced by more performant B-tree.
Querying
One needs to understand that with in-memory object indexes its only possible to implement subset of SQL query and no join queries. To start with, small framework only implements following operations : arithmetic operands : <,>,<=,>=,==. Logical operands : AND , OR. Range : BETWEEN, Set : IN
To implement querying SQL parser needs to be implemented. I choose to avoid writing parser and implemented direct API kind of querying similar to Quaere. I find Domain specific languages more intuitive since when developer writes code for it he know what querying he writes. So interface looks like below. It is not as good as quaere. Quaere is DSL, below API is just few interfaces implemented. Here execute returns List of IDs put into index against the properties.
So basically it is equivalent of follwing SQL
How it performs
Naturally is not as performant as Jofti since it used JDK tree map which uses red-black tree. In future when I complete writing my own b-tree implementation I expect to perform as good as Jofti. Jofti further implemented node level locking so multiple concurrent insert opertations can work parellel. This can also be implemented too. But query performance is not bad and I expect it to improve with b-tree implementation.
Integration with Terracotta
Since it uses TreeMap it is cluster-able easily. Attached tc-config.xml has all correct declarations for it. One more advantage is with Terracotta is that object identifier are readily generated by Terracotta. See implementation TerracottaQueryMap. Please dont compare performance of TerracottaQueryMap against HashMap, CHM or Terracotta Distributed Map(Concurrent String Map old name) since all these are just single index maps so easy to stripe or employ mulitple locks.
For using it as TIM you need to add following lines to tc-config.xml
In code you can use it as queryable map as follows. Here propList is list of properties to be indexed.
To Query.
Future directions planned
Java has hugh limitation for memory intensive application. So achieve scale two approaches : partition index and merge results or use disk to overflow index pages. Other thing I see can be implemented is that when query selects random elements these random elements need to be faulted from Terracotta server thus degrading performance, same elements can be read from local store too like EHCACHE. Later on this topic separately.
If you think this framework is useful please let me know. You can download its source as Eclipse project(sole dependency on tc.jar) and as Terracotta Integration Module here.
Tushar
querymap documentation link : Getting Started With Querymap
querymap source download : Complete Source Code with Eclipse Project
querymap TIM : Copy this TIM to terracotta modules directory to use it
querymap terracotta sample app : Download sample eclipse App. You can use Terracotta eclipse plugin to launch it within eclipse
So the basic idea is maintaining in-memory indexes which index String Ids with "Comparable" as Keys. All primitive wrappers in Java implement this interface so no special conversion is needed.
How it works
It scans java objects and makes comparable objects for each of the property mentioned to be indexed. It inserts these comparable objects in its own tree against String Ids assigned to the java object. So in-effect its nothing more than maintaining many in-memory Maps. In fact the implementation that I wrote uses JDK TreeMap and not b-tree map. But in-future it will be replaced by more performant B-tree.
Querying
One needs to understand that with in-memory object indexes its only possible to implement subset of SQL query and no join queries. To start with, small framework only implements following operations : arithmetic operands : <,>,<=,>=,==. Logical operands : AND , OR. Range : BETWEEN, Set : IN
To implement querying SQL parser needs to be implemented. I choose to avoid writing parser and implemented direct API kind of querying similar to Quaere. I find Domain specific languages more intuitive since when developer writes code for it he know what querying he writes. So interface looks like below. It is not as good as quaere. Quaere is DSL, below API is just few interfaces implemented. Here execute returns List of IDs put into index against the properties.
import static com.google.code.querymap.ObjectQuery.*;
Collection col = from(Domain.class).where(
gt("inner.property2", 60),
lt("inner.property2",89)
eq("inner.property3",random.nextInt(NUM_OBJECTs))
).execute();
So basically it is equivalent of follwing SQL
select from Domain
where inner.property2 > 60
and inner.property2 < property3=rand(NUM_OBJECTS)
How it performs
Naturally is not as performant as Jofti since it used JDK tree map which uses red-black tree. In future when I complete writing my own b-tree implementation I expect to perform as good as Jofti. Jofti further implemented node level locking so multiple concurrent insert opertations can work parellel. This can also be implemented too. But query performance is not bad and I expect it to improve with b-tree implementation.
Integration with Terracotta
Since it uses TreeMap it is cluster-able easily. Attached tc-config.xml has all correct declarations for it. One more advantage is with Terracotta is that object identifier are readily generated by Terracotta. See implementation TerracottaQueryMap. Please dont compare performance of TerracottaQueryMap against HashMap, CHM or Terracotta Distributed Map(Concurrent String Map old name) since all these are just single index maps so easy to stripe or employ mulitple locks.
For using it as TIM you need to add following lines to tc-config.xml
<modules>
<module id="com.google.code" name="querymap-1.0" version="1.0.0">
</module>
In code you can use it as queryable map as follows. Here propList is list of properties to be indexed.
TerracottaQueryMap map = new TerracottaQueryMap(Domain.class,proplist));
map.put(key,domain1);
map.put(key,domain2);
map.put(key,domain3);
To Query.
Collection col =map.entrySet(
from(Domain.class).
where(
eq("inner.property2",random.nextInt(NUM_OBJECTs))
)
);
Future directions planned
Java has hugh limitation for memory intensive application. So achieve scale two approaches : partition index and merge results or use disk to overflow index pages. Other thing I see can be implemented is that when query selects random elements these random elements need to be faulted from Terracotta server thus degrading performance, same elements can be read from local store too like EHCACHE. Later on this topic separately.
If you think this framework is useful please let me know. You can download its source as Eclipse project(sole dependency on tc.jar) and as Terracotta Integration Module here.
Tushar
Friday, July 10, 2009
Erlang and Concurrency
Here I write a lot about my experiments with Terracotta which is shared durable memory for Java applications which are multi-threaded. But some days before I came across erlang, language which is created at Erricsson for running fault-tolerant telecom applications. Whats so special about it?. If you google or rather "Bing" it you will find plenty of things about it. How cool and scalable it is?. If you read this you will even come to know How Goldman Sachs was using it to gain significant advantage in program trading over other competitors and how others are trying to steal leaked source code.
So erlang is not procedural programming language its functional programming language. Frankly I also need to understand whats so different about it. But I saw this presentation of infoQ site about erlang concurrency and was amazed. Right from starting I always thought about following graph - throughput increases as a function of in-coming request rate till some point but after it stabilizes and then it drops. It drops because of system overload. In a perfectly cpu-intensive lock-contention free application this will happen because of cpu context switching.
But in erlang it stays constant instead your latency (response time) increases. This is according to Little's Law. Little law says relation between throughput and latency is number of users in system. N = RX.
Above is famous graph of benchmark of YAWS (Http server written in erlang) against Apache and you can see how early apache gets saturated and dies. You can read details here but explanation is given here is :
"The problem with Apache is not related to the Apache code per se but is due to the manner in which the underlying operating system (Linux) implements concurrency. We believe that any system implemented using operating system threads and processes would exhibit similar performance. Erlang does not make use of the underlying OS's threads and processes for managing its own process pool and thus does not suffer from these limitations."
So basically all magic is erlang's concurrency model : No Shared State, Only message passing between light-weight processes. Erlang processes are way lighter than Java Threads since they are logical entities and not tied to user-level or kernel-threads. Thus erlang shows "No Shared State" concurrency model scales well. Since now JVM is touted as platform, I am looking forward to see erlang implementation on JVM and see how it does against other concurrent interpreted languages - scala. This is great post why JVM is unfit for such porting. May be Java 8 ( I think closures are not part of Java 7). This is also interesting read about Erlang on Java : Erlang Concurrency model on JVM . Some of work on writing OTP(erlang's sdk for writing applications) for scala http://github.com/jboner/scala-otp/tree/master/.
I have already got Programming Erlang book now looking forward to write first program in OTP.
Tushar
So erlang is not procedural programming language its functional programming language. Frankly I also need to understand whats so different about it. But I saw this presentation of infoQ site about erlang concurrency and was amazed. Right from starting I always thought about following graph - throughput increases as a function of in-coming request rate till some point but after it stabilizes and then it drops. It drops because of system overload. In a perfectly cpu-intensive lock-contention free application this will happen because of cpu context switching.
But in erlang it stays constant instead your latency (response time) increases. This is according to Little's Law. Little law says relation between throughput and latency is number of users in system. N = RX.
Above is famous graph of benchmark of YAWS (Http server written in erlang) against Apache and you can see how early apache gets saturated and dies. You can read details here but explanation is given here is :
"The problem with Apache is not related to the Apache code per se but is due to the manner in which the underlying operating system (Linux) implements concurrency. We believe that any system implemented using operating system threads and processes would exhibit similar performance. Erlang does not make use of the underlying OS's threads and processes for managing its own process pool and thus does not suffer from these limitations."
So basically all magic is erlang's concurrency model : No Shared State, Only message passing between light-weight processes. Erlang processes are way lighter than Java Threads since they are logical entities and not tied to user-level or kernel-threads. Thus erlang shows "No Shared State" concurrency model scales well. Since now JVM is touted as platform, I am looking forward to see erlang implementation on JVM and see how it does against other concurrent interpreted languages - scala. This is great post why JVM is unfit for such porting. May be Java 8 ( I think closures are not part of Java 7). This is also interesting read about Erlang on Java : Erlang Concurrency model on JVM . Some of work on writing OTP(erlang's sdk for writing applications) for scala http://github.com/jboner/scala-otp/tree/master/.
I have already got Programming Erlang book now looking forward to write first program in OTP.
Tushar
Thursday, July 9, 2009
Links : Java Sample Apps
Many times you hear or read something cool about some framework or tool and you are interested in Sample application written for it just to play with it, browse and go through source-code to find out how to code it.
Here is list of great sample Apps that I just stumbled upon while reading this great blog about Tomcat Clustering.
Link : http://www.mediafire.com/jbs-blog-examples
List is
* IBM-DB2-JDBC-XML.zip
* IBM-DB2-JDBC-Relational.zip
* Google-App-Engine.zip
* WebServices-JAX-WS-Java-SE.zip
* RESTful-WebServices-Apache-CXF.zip
* WebServices-Apache-CXF-Spring-2.5.zip
* WebServices-Apache-CXF.zip
* WebServices-WSIT-Reliable-Messaging.zip
* WebServices-JAX-WS-Web-App-Client-Basic-Security.zip
* WebServices-JAX-WS-Web-App-Client.zip
* WebServices-JAX-WS-Web-App.zip
* PMD-Clover2-Cobertura-Maven2-Test.zip
* WebServices-JAX-WS.zip
* WebServices-Axis2-with-Eclipse-client.zip
* Spring-Maven2-annotations-example.zip
* Spring-Maven2-basic-example.zip
I will add following to above lists which I know about
Terracotta Samples Application written by Team
Here is list of great sample Apps that I just stumbled upon while reading this great blog about Tomcat Clustering.
Link : http://www.mediafire.com/jbs-blog-examples
List is
* IBM-DB2-JDBC-XML.zip
* IBM-DB2-JDBC-Relational.zip
* Google-App-Engine.zip
* WebServices-JAX-WS-Java-SE.zip
* RESTful-WebServices-Apache-CXF.zip
* WebServices-Apache-CXF-Spring-2.5.zip
* WebServices-Apache-CXF.zip
* WebServices-WSIT-Reliable-Messaging.zip
* WebServices-JAX-WS-Web-App-Client-Basic-Security.zip
* WebServices-JAX-WS-Web-App-Client.zip
* WebServices-JAX-WS-Web-App.zip
* PMD-Clover2-Cobertura-Maven2-Test.zip
* WebServices-JAX-WS.zip
* WebServices-Axis2-with-Eclipse-client.zip
* Spring-Maven2-annotations-example.zip
* Spring-Maven2-basic-example.zip
I will add following to above lists which I know about
Terracotta Samples Application written by Team
- Terracotta Reference Application : Examinator
Terracotta with Spring and EHCACHE
Terracotta, Hibernate, Spring
Terracotta, Spring, Hibernate- Sample Web App with Maven, Terracotta
Monday, June 29, 2009
Terracotta's Hibernate Integration
This post is re-post of my earlier write-behind post but in different perspective : Terracotta's Hibernate integration 3.1
With version 3.1 Terracotta has implemented its own Caching for Hibernate Second Level Caching Provider. Earlier Terracotta's hibernate integration approach was : clustering EHCACHE. Terracotta with its JVM clustering ability, it was easily possible to cluster any POJO structure. So before 3.1, you might have used EHCACHE as hibernate second level cahce provider and tim-hibernate and tim-ehcache for clustering second level cache. With version 3.1 onwards terracotta will have its own cache backed by map-evictor and concurrent string map. Apart from this new hibernate integration has lots of new additions like cache admin console and read-write cache. Cache is always up-to-date and coherent.
But what I feel is that Terracotta platform is way more capable and following additional features can be added to make applications more scalable. These are just cool ideas.
Cache Warm-up feature
It would be nice feature to refresh or load cache whenever application or application cluster is starting up. This can easily be implemented with some sort of CacheLoader interface where Terracotta can callback this interface when faulting cache objects from terracotta server during first access. But such warm-up is only required on full cluster restart otherwise lot of meaningful cache entrites will get overwritten.
Write-Behind Caching
When you think of cache you will arrive at these cache strategies : Read-Through Caching, Write-Through Caching, Write-Behind Caching. Hibernate Second Level cache is Read-Write-Through Cache where if cache miss occurs, entity is read from database and then handed over to cache for susequent access. But H2LC is not Write-Behind caching. With Terracotta's disk persistence and asynchronsous module it would be really efficient for certain use-cases to implement write-behind. Currently hibernate just directly writes to database. Instead if its modified to write to second level cache and persistent async-database-queue, this would decrease latency and increase throughput dramatically. Imagine if you can schedule all your database writes in non-business hours using tim-async. I find write-behind is certainly the best way to reduce pressure on database. And with Terracotta's clusterwide coherent persistent datastore its practicaly possible. Terracotta would be your database guard taking all your querying as well as database inserts on its shoulders.
But this model would require certain changes in the way hibernate works. especially query cache. Since now Terracotta will have latest snapshot of yor System of Record, queries have to be executed against cache and not database. Thus it can not be generic solution. You can implemented write-behind only in certain cases where your business use case permits it. On the other hand to solve query problem Querymap that i disucssed in my previous posts can be used to query certain type of data. So if your business use case permits write-behind and query-map can give you very fast database accelerator. In one of my previous jobs I was working on financial application where certain set of objects were modified at very high rate and same were queried against. For such application classic replicated H2LC does not bring any value, instead it will degrade the performance due to overhead during frequent-cluster-wide updates. But Terracotta will make it scalable, forwarding updates only to Node on which cache entry exists, updating the object clusterwide so when AsyncProcessor picks it up it will contain all the changes made. Its Terracotta's DSO Magic.
Advantage here is that you dont have to do religious shift of Killing Your Database Totally. Database is your System of Record. With Terracotta Hibernate Accerlerator you are only delaying updates to SOR and not replacing it.
Currently I am going through Hibernate source code and learning how hiberante event mechanism works. My guess is that write-behind can be implemented with hibernate events. If not I may try to modify the source code to add write-behind and h2lc-cache querying capability. Hibernate search is similar where instead of classic session you get Indexing-aware session.
With Terracotta FX (assuming your application requires more than 4000 write operations per second - avg throughput of one un-tuned Terracotta server) your write throughput will increase linearly which is not possible with any RDBMS on any type of hardware.
I hope Terracotta will add these features in coming versions. Terracotta 3.1 Hibernate Integration is just start.
With version 3.1 Terracotta has implemented its own Caching for Hibernate Second Level Caching Provider. Earlier Terracotta's hibernate integration approach was : clustering EHCACHE. Terracotta with its JVM clustering ability, it was easily possible to cluster any POJO structure. So before 3.1, you might have used EHCACHE as hibernate second level cahce provider and tim-hibernate and tim-ehcache for clustering second level cache. With version 3.1 onwards terracotta will have its own cache backed by map-evictor and concurrent string map. Apart from this new hibernate integration has lots of new additions like cache admin console and read-write cache. Cache is always up-to-date and coherent.
But what I feel is that Terracotta platform is way more capable and following additional features can be added to make applications more scalable. These are just cool ideas.
Cache Warm-up feature
It would be nice feature to refresh or load cache whenever application or application cluster is starting up. This can easily be implemented with some sort of CacheLoader interface where Terracotta can callback this interface when faulting cache objects from terracotta server during first access. But such warm-up is only required on full cluster restart otherwise lot of meaningful cache entrites will get overwritten.
Write-Behind Caching
When you think of cache you will arrive at these cache strategies : Read-Through Caching, Write-Through Caching, Write-Behind Caching. Hibernate Second Level cache is Read-Write-Through Cache where if cache miss occurs, entity is read from database and then handed over to cache for susequent access. But H2LC is not Write-Behind caching. With Terracotta's disk persistence and asynchronsous module it would be really efficient for certain use-cases to implement write-behind. Currently hibernate just directly writes to database. Instead if its modified to write to second level cache and persistent async-database-queue, this would decrease latency and increase throughput dramatically. Imagine if you can schedule all your database writes in non-business hours using tim-async. I find write-behind is certainly the best way to reduce pressure on database. And with Terracotta's clusterwide coherent persistent datastore its practicaly possible. Terracotta would be your database guard taking all your querying as well as database inserts on its shoulders.
But this model would require certain changes in the way hibernate works. especially query cache. Since now Terracotta will have latest snapshot of yor System of Record, queries have to be executed against cache and not database. Thus it can not be generic solution. You can implemented write-behind only in certain cases where your business use case permits it. On the other hand to solve query problem Querymap that i disucssed in my previous posts can be used to query certain type of data. So if your business use case permits write-behind and query-map can give you very fast database accelerator. In one of my previous jobs I was working on financial application where certain set of objects were modified at very high rate and same were queried against. For such application classic replicated H2LC does not bring any value, instead it will degrade the performance due to overhead during frequent-cluster-wide updates. But Terracotta will make it scalable, forwarding updates only to Node on which cache entry exists, updating the object clusterwide so when AsyncProcessor picks it up it will contain all the changes made. Its Terracotta's DSO Magic.
Advantage here is that you dont have to do religious shift of Killing Your Database Totally. Database is your System of Record. With Terracotta Hibernate Accerlerator you are only delaying updates to SOR and not replacing it.
Currently I am going through Hibernate source code and learning how hiberante event mechanism works. My guess is that write-behind can be implemented with hibernate events. If not I may try to modify the source code to add write-behind and h2lc-cache querying capability. Hibernate search is similar where instead of classic session you get Indexing-aware session.
With Terracotta FX (assuming your application requires more than 4000 write operations per second - avg throughput of one un-tuned Terracotta server) your write throughput will increase linearly which is not possible with any RDBMS on any type of hardware.
I hope Terracotta will add these features in coming versions. Terracotta 3.1 Hibernate Integration is just start.
Monday, May 25, 2009
Querying Java Objects stored in Terracotta's NAM Part 2
First part of this series, I talked about existing frameworks and what I found out is that they lack indexing hence rarely useful for large data-sets. So I tried finding out how to do indexing. My idea was simple : index objects and store reference to object in index then with Terracotta you can cluster objects and index as well. So it becomes "queryable" datastore. My first attempt was to find out how indexing is done. By book it says b-tree index. I found out this(jdbm) framework which is trying to do the persistent DB in Java by implementing B-tree indexes on disk.I took only b tree and implemented simple query parser. What it does is that it traverses b-tree and finds out tuples and then returns them.
After this first attempt, then I experimented with Lucene. Lucene is not tree-index, its inverted index but it has lot of capability and its fast, supports in-memory and disk-based indexes.
So here is my little framework for queryable datastore :
public interface TCQueryMap
void init();
Map
}
Naturally its extension of Map which is single index. My implementation wraps a HashMap with ReadWrite locks and Lucene RAMDirectory index. So all get/puts hit index within lock boundaries and then you can query index. This is very simple, I have not gone into complexities like spill-over of index onto disk etc.
LuceneIndexingConfig config = new LuceneIndexingConfig();
List
// index three properties only
propList.add("accountName");
propList.add("person.age");
propList.add("person.name.firstName");
config.setIndexPropertyList(propList);
TCQueryMap indexer = new LuceneQueryStore(config);
// add object
Acccount account .....
indexer.put("user99",account);
// query object
Map col = indexer.query("person.age:21");
I also came to know about Jofti from one of comments. Jotfi is what I would eventually like to write. I don't know why its not used by many people. One reason could be its not maintained. I found it pretty useful so I plugged in Jofti as well in my framework.
Now lets compare it with simple Hibernate-JDBC based solution. Obviously its not perfect. SQL is way more complex and expressive language. But here we are talking about cached data and I am sure once data is cached ( it means objectified from join query on relational DB) very few times you will require join, its mostly "where clause" of one or more conditions. Lucene does that very well.
So lets see numbers. I have not done any tuning apart from standard lucene stuff. One of main parameters is how many properties you want to index. This determines index size, memory and speed.
Below is small benchmark showing 60K objects inserts with three properties indexed and then random queries on three properties
Lucene Inserts/Sec = 1000
Jofti Inserts/Sec = 8793.78
Lucene Queries/Sec = 5172
Jofti Queries/Sec =13636.36
Results for 14 properties indexed :
Lucene Inserts/Sec = 740
Jofti Inserts/Sec = 4866.96
Lucene Queries/Sec = 3750
Jofti Queries/Sec = 12500
Since Jofti is Tree index it outperforms Lucene Index. The problem with Lucene is that once index gets bigger insert performance slows down. Also these numbers are taken with one commit on one put operation. If you index lot of objects together and then commit, lucene is also fast, that's how it is to be used - Batch API. On the other hand Jotfi is fast, I could not find any details about being thread-safe and other concurrency issues so I wrapped it around Lock. I don't know why Jofti is not used by many people.
Also what if you can run Hibernate/JPA queries on Map? that would be great. Its already done by hibernate team. They run query against Second level cache but it would big task to find out and extract idea out of it. Just a thought. Second Level Query cache gets invalidated when you modify single entity, imagine if we update the same object in QueryMap cache you dont need Query cache of course querying capability is not great.
Another thought that comes in my mind is clustering in-memory databases like H2 or HSQLDB. Imagine the benefits of it. But then its anti-terracotta. Why? it would be Relational DB with baggage of ORM mismatch.
Entire source code you can download it from here : http://code.google.com/tc-querymap/. Tar file is just bunch of java files and its very early
prototype. Stay tuned to project, I will update it once I finish with proper integration with Terracotta.
So if you find it useful please leave comments, I would love to hear from you.
Monday, May 4, 2009
Got one!!
Finally I got my own I Love Linux T-shirt, here is snap.
You too can create your own T-shirt with Geeky quotes or any other quotes or picture. I am planning to make another with Ubuntu but waiting/searching for good image apart from standard "Linux for Human Beings"
You too can create your own T-shirt with Geeky quotes or any other quotes or picture. I am planning to make another with Ubuntu but waiting/searching for good image apart from standard "Linux for Human Beings"
Wednesday, April 15, 2009
Portable Ubuntu Rocks
Years ago (literally 2.5 years ago) I had tried co-linux. At that time it was in initial stages but was working perfectly in text mode. If you don't know whats co-linux, its linux distribution which works like Windows Binary. No need to setup Virtaul Machine Emualator and install or add virtual images. This was when I had never heard of Virtualization and I was really amazed of the idea. At that time co-linux had managed some elementary GUI drawing mainly KDE applications (at least i had seen screeshots), it did not work though on my machine. Just minutes before I downloaded portable ubuntu after reading this Post from LifeHacker. And it works!! just like described. Who needs VMWare and stuff like that if you can work on linux shell as well as Firefox just Alt-Tab apart. Here is screenshot of Portable Ubuntu in action.
Saturday, April 11, 2009
Links : List of Geeky Quotes
Here : http://rafaeltrindade.com.br/2008/09/21/best-geek-quotes/
My Fav is I would love to change the world, but they won’t give me the source code
My Fav is I would love to change the world, but they won’t give me the source code
Wednesday, April 8, 2009
Links : Distributed Hash Tables
Nice List of Distributed Key-Value Stores : http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/
Querying Java Objects stored in Terracotta's NAM
This post is inspired from : http://forums.terracotta.org/forums/posts/list/1965.page . Nothing new .. just another word in blogosphere.
Terracotta is gr8 clustering solution in-fact its platform-level service hence has large number of uses. One of the use is using it as database. Terracotta can never replace database but it can play role of data storage media very well. One of major disadvantage is lack of querying data. Only way you can query data is Map. Map is like single index so if you want to get list of object satisfying some criteria you are required to integrate through entire collection. There are already APIs written for querying java collections. So you can use them with Terracotta NAM.
When you think about querying there are lot of factors : Query Language, Its Performance - Optimizers, Operations Supported : Select, Update, Delete, Joins
JoSQL is good API for querying java collections with good documentation. I did small test with 1 million objects and random query took around 800ms which is way too much. Again its simple iteration through collection due to lack of indexes and query execution plan. Problem with Indexes is that Object graphs can change and at every change you are required to recompute the index which would be difficult to do : as complex as Terracotta's bytecode instrumentation. You can find test code here
Query Language : Moderately good, Performance : Not good for large collections, No update or delete only select projection queries. No joins
Quaere is a very flexible DSL that lets you perform a wide range of queries against any data structure that is an array, or implements the java.lang.Iterable. Its sort of port of LINQ of .Net world. I think linq is next generation data quering tool -cleaner. Quaere is not query language but query API just like Hiberate Criteria query but more elegant. I really liked quaere - its really powerful its support join operation. You can read this post for details : Solving Puzzles with Quaere Its still beta level and not released. One of Queare's another sister project is its JPA integration. Imagine you could write standard JPA application with Quaere as query language and Terracotta as persistent store. No need of database. But as with JoSQL Quaere is also slow. I mean slower than RDBMS. I did small test with 1 million object similar to JoSQL test and response time was similar to JoSQL. You can download test code here
This post also discusses jmap's OQL implementation. It uses rhino javascript engine behind with hashtables.I did consider it to port for Terracotta but its custom written for Object Heap Dumps. JxPath is another tool with which you can query java collections using XPath expressions. I did not evaluate JxPath since i felt it will be on similar lines of JoSQL and Quaere, only different flavor. If you have used XPath earlier then this is much easier to use.
GlazedList is event driven list API specially designed for Swing Applications displaying table and list data. But if you consider List of Objects as table (each object is row and its properties as columns) a proper in-memory index can be maintained for querying. But this applies to only root object level. What if inner object in object graph store in your container changes?. You may then need to update the container whenever object changes. So i guess maintaining in-memory index for java objects is pretty difficult thing to do.
With such tools i think you can easily query moderate size java collections stored in Terracotta's durable memory with acceptable response time.
Tushar
Terracotta is gr8 clustering solution in-fact its platform-level service hence has large number of uses. One of the use is using it as database. Terracotta can never replace database but it can play role of data storage media very well. One of major disadvantage is lack of querying data. Only way you can query data is Map. Map is like single index so if you want to get list of object satisfying some criteria you are required to integrate through entire collection. There are already APIs written for querying java collections. So you can use them with Terracotta NAM.
When you think about querying there are lot of factors : Query Language, Its Performance - Optimizers, Operations Supported : Select, Update, Delete, Joins
JoSQL
JoSQL is good API for querying java collections with good documentation. I did small test with 1 million objects and random query took around 800ms which is way too much. Again its simple iteration through collection due to lack of indexes and query execution plan. Problem with Indexes is that Object graphs can change and at every change you are required to recompute the index which would be difficult to do : as complex as Terracotta's bytecode instrumentation. You can find test code here
Query Language : Moderately good, Performance : Not good for large collections, No update or delete only select projection queries. No joins
Quaere
Quaere is a very flexible DSL that lets you perform a wide range of queries against any data structure that is an array, or implements the java.lang.Iterable. Its sort of port of LINQ of .Net world. I think linq is next generation data quering tool -cleaner. Quaere is not query language but query API just like Hiberate Criteria query but more elegant. I really liked quaere - its really powerful its support join operation. You can read this post for details : Solving Puzzles with Quaere Its still beta level and not released. One of Queare's another sister project is its JPA integration. Imagine you could write standard JPA application with Quaere as query language and Terracotta as persistent store. No need of database. But as with JoSQL Quaere is also slow. I mean slower than RDBMS. I did small test with 1 million object similar to JoSQL test and response time was similar to JoSQL. You can download test code here
This post also discusses jmap's OQL implementation. It uses rhino javascript engine behind with hashtables.I did consider it to port for Terracotta but its custom written for Object Heap Dumps. JxPath is another tool with which you can query java collections using XPath expressions. I did not evaluate JxPath since i felt it will be on similar lines of JoSQL and Quaere, only different flavor. If you have used XPath earlier then this is much easier to use.
GlazedList is event driven list API specially designed for Swing Applications displaying table and list data. But if you consider List of Objects as table (each object is row and its properties as columns) a proper in-memory index can be maintained for querying. But this applies to only root object level. What if inner object in object graph store in your container changes?. You may then need to update the container whenever object changes. So i guess maintaining in-memory index for java objects is pretty difficult thing to do.
With such tools i think you can easily query moderate size java collections stored in Terracotta's durable memory with acceptable response time.
Tushar
Tuesday, April 7, 2009
Maven : Java Profiling
At workplace I use maven extensively. In fact I like maven so much that I am slowly moving lot of java eclipse project into maven project. Maven has very good eclipse plugin which makes integration with Eclipse very easy. You can run maven from eclipse. Maven is really good tool. If you have any specific needs you can write your own plugin. At my previous workplace, senior engineer in my project automated everything (starting jboss servers on remote machine in various environments : testing, perf-testing, integration testing etc. database population, database schema drop/rebuild etc).
But now I wanted to profile java application which I used to run from maven. One of the best part of maven is dependency management and repository - it builds classpath automatically for you, but then its pain too. If you want to run java application you have to do through Maven. There is exec plugin with which you can run any Application or shell script and there is exec:java with which you can run java Main class. The problem is that exec:java is in same JVM. So you cant run any java agent(-agent) or other things : Specifically Java Profiling. You should make sure you run the application within the same environment/settings.
So my first task was to get the complete classpath and then launch java with profiler java options. I am using jprofiler which uses JVMTI agent hence you need to append "-agentlib:jprofilerti=port=31757 -Xbootclasspath/a:/Applications/jprofiler5/bin/agent.jar" to java command line.
Here is little shell script through which I managed to do java profiling for maven Project. This will work only for J2SE applications tough!. Frustrating part was variable DYLD_LIBRARY_PATH. I was new to MacOS and was trying with usual LD_LIBRARY_PATH and -agentpath jvm option. Surprisingly -agentpath option should work on MacOS but it didnt work i guess some problem with Jprofiler binary. But lastly i managed to profile my application properly.
This is the first time I had to do away with maven command. I wished somebody had written maven plugin for lauching jprofiler enabled apps. There is maven plugin for Yourkit Java Profiler : http://code.google.com/p/maven-yourkit-plugin/ but it did now work.
But now I wanted to profile java application which I used to run from maven. One of the best part of maven is dependency management and repository - it builds classpath automatically for you, but then its pain too. If you want to run java application you have to do through Maven. There is exec plugin with which you can run any Application or shell script and there is exec:java with which you can run java Main class. The problem is that exec:java is in same JVM. So you cant run any java agent(-agent) or other things : Specifically Java Profiling. You should make sure you run the application within the same environment/settings.
So my first task was to get the complete classpath and then launch java with profiler java options. I am using jprofiler which uses JVMTI agent hence you need to append "-agentlib:jprofilerti=port=31757 -Xbootclasspath/a:/Applications/jprofiler5/bin/agent.jar" to java command line.
Here is little shell script through which I managed to do java profiling for maven Project. This will work only for J2SE applications tough!. Frustrating part was variable DYLD_LIBRARY_PATH. I was new to MacOS and was trying with usual LD_LIBRARY_PATH and -agentpath jvm option. Surprisingly -agentpath option should work on MacOS but it didnt work i guess some problem with Jprofiler binary. But lastly i managed to profile my application properly.
mvn dependency:build-classpath -Dmdep.outputFile=mycp.txt
export CLASSP=`cat ./mycp.txt`
export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/Applications/jprofiler5/bin/macos
$JAVA_HOME/bin/java -cp $CLASSP:./target/classes -agentlib:jprofilerti=port=31757 -Xbootclasspath/a:/Applications/jprofiler5/bin/agent.jar $*
This is the first time I had to do away with maven command. I wished somebody had written maven plugin for lauching jprofiler enabled apps. There is maven plugin for Yourkit Java Profiler : http://code.google.com/p/maven-yourkit-plugin/ but it did now work.
Monday, March 9, 2009
Links : What is REST?
I needed to learn about REST interface and wanted some good article for learning basics of REST : REpresentational State Transfer. This article : A Brief Introduction to REST is really the best one I found so thought of sharing it with you. REST is really nice idea of further simplifying the modeling of web-applications. I implemented by REST program with JSON and my client is in Javascript which is really cool (People are writing entire web based OS and here i am playing with simple java script stuff :-) ) With help of java-script libraries you can easily achieve data binding : like FLEX and other RIA. I really enjoyed working with this totally different stuff. I will add snippets in coming days.
..
Tushar
..
Tushar
Thursday, February 26, 2009
Links : Things I Wish I’d Been Told
When I look back in my engineering days and now, I certainly figured out one thing : Whatever they teach in class hardly matters - the syllabus is totally outdated. What matters is that how you learn new things quickly, be updated in this fast moving world of technology.
Here I am sharing a link I Wish I’d Been Told, when I graduated with computer science engineering degree. Tips For Students with a Bachelors in Computer Science
Here I am sharing a link I Wish I’d Been Told, when I graduated with computer science engineering degree. Tips For Students with a Bachelors in Computer Science
Subscribe to:
Posts (Atom)