Some Messages Stuck in the ActiveMQ Queue

In one of our product we are using Apache ActiveMQ 5.5.0 with Spring 3.0.7.  I have two publisher pushing message to a common queue and two consumers listening from the same common queue.
Last week I encountered this strange issue where some messages got stuck in the queue (I could see the message in the ACTIVEMQ_MSGS table since I am using persisted queue).
The strange part was only few messages were getting stuck while others were still getting processes just fine.

I looked into the logs and started thinking that I am hitting some ActiveMQ bug (possibly https://issues.apache.org/jira/browse/AMQ-3966). But I continued my diagnosis. Today after spending about two days on this issue I realized that it was NOT ActiveMQ bug but a bug in my code.

Here is what was happening, for some of the messages the code was making an HTTP call to a REST API. Those calls never got completed and just got blocked. Since the consumer had 10 threads to handle messages , so even when one thread got stuck others were still working fine. But slowly even these threads got stuck as they receive similar message and tried to make the HTTP call the same REST API. And finally the consumer stopped processing messages.
Interestingly, this behavior was happening only on one of the consumer and the other consumer was able to make HTTP calls the REST API successfully.

From ActiveMQ brokers point of view both of the consumers are up and running , so it keep on sending half of the message to the first consumer and hence all the those message get piled up in the ACTIVEMQ_MSGS table.

Restarting the consumer resolved the issue because then the threads were re-created. And by that time the issue with making API calls was also resolved.

So, the learning from this issue is, I should have added some READ_TIMEOUT to the HttpClient while making the REST API call. That way the thread would have thrown the "READ_TIMEOUT" error and got freed to process next message.

Hope this will help someone.

Enable Conversation using Session attributes in Spring

In a Spring framework project we use form objects saved as session attributes to achieve a conversational style of creating and editing business objects.
But recently I realized that this functionality does not work if you have more than one tab open in the same browser. Reason is simple, if you are editing an object in first tab and start editing another object in second tab then the session attributes gets replaced by the second object. And now if you save first object then actually the second object will be updated with the information of first object.

This happens because spring saves the objects into session with same attribute name, so the object saved later will replace any other object. And when POST request is made from already loaded UI then it will always update the object which was saved later.

There is a very easy solution to the above issue. We can extend the class "DefaultSessionAttributeStore" and override just one method,  which is "getAttributeNameInSession(WebRequest request, String attributeName)", as shown below:

  
@Override
  protected String getAttributeNameInSession(WebRequest request, String attributeName) {
    String cid = request.getParameter(attributeName + "_cid") == null ? ""
        + request.getAttribute(attributeName + "_cid", WebRequest.SCOPE_REQUEST) : request.getParameter(attributeName
        + "_cid");
    if (cid != null || !"".equals(cid)) {
      return super.getAttributeNameInSession(request, attributeName + "_" + cid);
    }
    return super.getAttributeNameInSession(request, attributeName);
  }

This class should also implement interface "InitializingBean" and override method "afterPropertiesSet()" as shown below:

  @Override
  public void afterPropertiesSet() throws Exception {
    annotationMethodHandlerAdapter.setSessionAttributeStore(this);
  }

This will make sure that this custom session attribute store is added to the annotation handler adaptor.

Now when ever you save the form into model map, make sure that you add another attribute with name "{your form name}_cid" and value as the unique id for the form.
And from the JSP add a hidden input which will be sent along with the POST request.

<input name="{your form name}_cid" type="hidden" value="<c:out value='${your form unique id}'/>" />

Thats it! You can now edit different entities in different tabs under same browser session.

Please add comment if you have any question.

Thanks,
Manish

Resolving "Could not resolve view with name ... in servlet with name ..."

I was creating a web application using Spring with Apache Tiles, and my application did not work after I enabled the TilesView and TilesConfigurer. My configuration was as shown below:



I was getting below error.
SEVERE: Servlet.service() for servlet [onecode] in context with path [/onecode] threw exception [Could not resolve view with name 'search' in servlet with name 'onecode'] with root cause
javax.servlet.ServletException: Could not resolve view with name 'search' in servlet with name 'onecode'
    at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1200)
    at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1005)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:952)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

From the error it seems the control is not even reached to Tiles because I did not see any Tiles class in the stack trace. I checked all the configuration files multiple times and everything looked just fine.
After spending some time struggling to find the issue, I noticed that I had a typo in the "tiles-def.xml" file. One of the definition was extending a base definition and there I typed the incorrect name of the extended definition. The issue got resolved as soon as I corrected the base definition.

~Manish

Jail breaking Cognos: Fix ClickJacking in Cognos

All of us who work with Cognos, know that sometimes how difficult it can be to fix or customize a simple request from the End user, for example modifying Prompt behaviors.

I’ve faced my share of Fancy requests like Hiding the Prompt name from the Drop-down <ToDo> or Reload a prompt without Refreshing the page.

Thanks to experts like “Cognos Paul” and groups like Ironside and Cognoise, we know how to get these done.

This time I got a little more complex request than UI customizations. In the security testing for our application, Testing team reported that Cognos is susceptible to  ClickJacking (or Frame busting).

IBM replied with “you can configure your Web-Server to set X-FRAME-OPTIONS that disables framing”.

But this works  only with latest browsers and if the victim is using a really old browser, like Mozilla 3.20 in our case, then it doesn’t.

There is a simple Solution to this, provided by OWASP, Link:

Add the following code to the landing page of your application.

Now comes the second part of the problem. In our application we are using Cognos LDAP authentication and hence don’t have a customized page for Login.

So the solution was to find Where to add this piece of code in Cognos so that the entire portal is secured from this issue.

With some effort I figured out that there are two pieces of the puzzle. First is that Cognos generates the HTML from its XSL files at runtime, so we can’t directly paste the code in the HTML.

Second was that there are some JS files, which are loaded for every page, as required by the portal.

So I found that for Login page, HTML is generated from this “render.xsl” file and for Landing/Portal pages, “framework.xsl”.

Then I added the above code to these two files like this:

For Login page:

For Portal pages:

You have to do it in two places as “framework.xsl” didn’t allow us to add HTML “Style” element, which then has to be added to  “presentation.xsl”

For “framework.xsl”

If done correctly, any webpage which tries to use your portal in an iframe, your portal will bust out of that frame and URL will be changed to actual portal URL.

Hope it helps.

Incorrect position of form object in spring controller

Today I wasted some time to find out why my spring controller method errors out even before entering into the method.
My failing method is:


But I knew I am doing things right. I debugged  the application and then searched on net but could find why it is failing. Suddenly I thought is it the position of the "User" model attribute which is causing the issue. Then I changed the method as below:


Notice that I moved the method argument "@ModelAttribute @Valid User user" from first to second last. And Voila!! it worked.

FYI: I am using Spring 3.2.

Learning MongoDB

I spent sometime to learn some basic stuff about MongoDB. I started searching internet and found some very helpful links.
Try the MongoDB online (interactive tutorial): http://try.mongodb.org/
Little MongoDB book (PDF) : http://openmymind.net/mongodb.pdf

After going through above links I got some confidence and I downloaded the MongoDB from http://www.mongodb.org/downloads .  Installing the db was very simple and within minutes I was having mongo up and running on my Macbook.

Then I thought of creating a sample spring application to learn spring integration with mongo. I followed the documentation at http://docs.spring.io/autorepo/docs/spring-data-mongodb/1.4.3.RELEASE/reference/html/index.html.

I created an application which fetches all users from the database. The code for the sample application is available at https://github.com/itsmanishagarwal/SpringMongoDb.

Anybody can download the code and use as per their need.

~Manish

Why should we allow only single session per user?

Some time back me , Mishra and Mittal were discussing some technical issues and then Mishra mentioned that he had a requirement to implement this feature to allow only one session per user. He implemented this without any problem but then we started thinking what are the possible reasons for which people want to implement this requirement. After some thinking, discussing and googling we could come with below reasons:

1. Security. If we allow only one session per user then on creation of second session we can alert the user that there is already a active session and allow a way to kill the previous session. This way if the user is unaware of previous session then this will serve as a warning that his/her credentials may have been compromised. And user can change his credentials.
2. Licensing. Some products are priced as per number of users using the product. So, avoiding the multiple session per user will prevent the misuse of the license.
3. Product Implementation. This is very specific to the product requirement. If the application maintains some kind of user's working state then multiple sessions can mess it up.

 I will update this post if I could find some more reasons.

~Manish

Unable to add any directory into the watched directory chain in virgo tomcat server 3.0.3

Recently I had a requirement where I had to add a directory into the watched directory chain in virgo tomcat server (VTS 3.0.3). Requirement was to check if some jars are present in a particular directory, and if jars are present then add the directory programmatically (using shell script) in the watched directory chain.
So, I created a shell script which looks like below:

if [ -f /usr/local/vts/external_jars/util* ] ; then

  echo "Found the util jar under /usr/local/vts/external_jars"

sed  -i '

  /usr.watchDirectory=repository\/usr/ a\

external_jars.type=watched \

external_jars.watchDirectory=external_jars

       ' $VTS_HOME/config/org.eclipse.virgo.repository.properties

sed -i "s/usr,/usr,external_jars,/" $VTS_HOME/config/org.eclipse.virgo.repository.properties

fi

When I ran the script it worked all fine, and the configuration file also got updated, but when I restarted my virgo server it did not pick my "external_jars" directory into the chain of watched directory.
I compared the changes with other watched directories but could not find any clue.

Then after struggling a lot I realized that there is a extra space after the value of property "external_jars.type". And as soon as I removed the extra space, it worked.

Corrected script is:

if [ -f /usr/local/vts/external_jars/util* ] ; then

  echo "Found the util jar under /usr/local/vts/external_jars"

sed  -i '

  /usr.watchDirectory=repository\/usr/ a\

external_jars.type=watched\

external_jars.watchDirectory=external_jars

       ' $VTS_HOME/config/org.eclipse.virgo.repository.properties

sed -i "s/usr,/usr,external_jars,/" $VTS_HOME/config/org.eclipse.virgo.repository.properties

fi

Just thought of sharing this info, so I blogged it.

How to quickly setup ActiveMQ broker, publisher and subscriber

Last week I was working on an issue related to ActiveMQ messaging. During my debugging the most painful part was to start the entire application and then execute the test scenario just to test some functionality or feature in ActiveMQ. After spending some time I realized that to speed up my debugging and anaysis I have to create a separate application/program which I can start and stop quickly after making changes. So, I created two programs:
ActiveMQPublisherTest: This program start the ActiveMQ broker and then push messages into a queue.
ActiveMQSubscriberTest: This program listens to the ActiveMQ broker started by "ActiveMQPublisherTest" and receives the event published by it.

I have published the entire source code on GitHub at: https://github.com/itsmanishagarwal/ActiveMQTest

To use these programs you just need to change the XML files to point to your database.

Any suggestions or feedback is welcome.

~Manish

How to verify if a file which belongs to an RPM is modified?

Recently I had to struggle to find out a way to verify if a file which belongs to an RPM is modified or not. After searching a bit on google I found that there is a option in "rpm" tool to verify all the files but there is no direct way to find if a particular file is modified. So, I decided to create a function which can help me doing that.

function isFileModified {

  FILE=$1

  if rpm -Vf $FILE | grep $FILE >/dev/null 2>&1 && rpm -Vf $FILE | grep $FILE \

| awk -F" " '{print $1}' | grep -e ".*5.*" >/dev/null 2>&1; then

    return 0

  else

    return 1

  fi

}

Explanation:
rpm -Vf $FILE : Returns list of all the files which got modified in the RPM package.
grep $FILE : Check if the file to be checked is in the list of modified file.
awk -F" " '{print $1}' : Truncates the attributes of the provided file
 grep -e ".*5.*"  : Check if the md5 digest of the file is changed.

So, the functions returns 0 if the file's md5 digect is changed after it is installed by the RPM. Else it will return 0.

Thanks,
Agry