In one of our product we are using Apache ActiveMQ 5.5.0 with Spring 3.0.7. I have two publisher pushing message to a common queue and two consumers listening from the same common queue.
Last week I encountered this strange issue where some messages got stuck in the queue (I could see the message in the ACTIVEMQ_MSGS table since I am using persisted queue).
The strange part was only few messages were getting stuck while others were still getting processes just fine.
I looked into the logs and started thinking that I am hitting some ActiveMQ bug (possibly https://issues.apache.org/jira/browse/AMQ-3966). But I continued my diagnosis. Today after spending about two days on this issue I realized that it was NOT ActiveMQ bug but a bug in my code.
Here is what was happening, for some of the messages the code was making an HTTP call to a REST API. Those calls never got completed and just got blocked. Since the consumer had 10 threads to handle messages , so even when one thread got stuck others were still working fine. But slowly even these threads got stuck as they receive similar message and tried to make the HTTP call the same REST API. And finally the consumer stopped processing messages.
Interestingly, this behavior was happening only on one of the consumer and the other consumer was able to make HTTP calls the REST API successfully.
From ActiveMQ brokers point of view both of the consumers are up and running , so it keep on sending half of the message to the first consumer and hence all the those message get piled up in the ACTIVEMQ_MSGS table.
Restarting the consumer resolved the issue because then the threads were re-created. And by that time the issue with making API calls was also resolved.
So, the learning from this issue is, I should have added some READ_TIMEOUT to the HttpClient while making the REST API call. That way the thread would have thrown the "READ_TIMEOUT" error and got freed to process next message.
Hope this will help someone.
Last week I encountered this strange issue where some messages got stuck in the queue (I could see the message in the ACTIVEMQ_MSGS table since I am using persisted queue).
The strange part was only few messages were getting stuck while others were still getting processes just fine.
I looked into the logs and started thinking that I am hitting some ActiveMQ bug (possibly https://issues.apache.org/jira/browse/AMQ-3966). But I continued my diagnosis. Today after spending about two days on this issue I realized that it was NOT ActiveMQ bug but a bug in my code.
Here is what was happening, for some of the messages the code was making an HTTP call to a REST API. Those calls never got completed and just got blocked. Since the consumer had 10 threads to handle messages , so even when one thread got stuck others were still working fine. But slowly even these threads got stuck as they receive similar message and tried to make the HTTP call the same REST API. And finally the consumer stopped processing messages.
Interestingly, this behavior was happening only on one of the consumer and the other consumer was able to make HTTP calls the REST API successfully.
From ActiveMQ brokers point of view both of the consumers are up and running , so it keep on sending half of the message to the first consumer and hence all the those message get piled up in the ACTIVEMQ_MSGS table.
Restarting the consumer resolved the issue because then the threads were re-created. And by that time the issue with making API calls was also resolved.
So, the learning from this issue is, I should have added some READ_TIMEOUT to the HttpClient while making the REST API call. That way the thread would have thrown the "READ_TIMEOUT" error and got freed to process next message.
Hope this will help someone.
