Memory leak in connection.

[chameleon]

I've had this problem for a long time now since first using Red5. After a random duration (usually within 2 hours with server at full load) the heap will grow rapidly and the garbage collection routines are unable to reduce the memory even after a full collection. This continues for about a minute until the server throws out of memory exceptions and quits working. I believe the issue to be related to one or more connections that continues to somehow consume resources. One thing that leads to me to think this is the random nature and having it suddenly occur rather than building up over time as it would if it was a server wide issue. Also if I try to issue a disconnect to every connection on the server one or more will block on the disconnect() call indefinitely.

The problem use to be more sporatic under lighter loads but now with a full load of maybe 5 connections per second sustaining an average of 200 connection constantly it happens much more often. That too kind of indicates it's likely one or more single connections causing it. I'll report anything else I can find but I've been trying to track this down for a long time now without much luck. So you know as well the server is used almost solely for VOD content and a couple live streams.

[nancsi]

Similar problem in my app, but not the same result.
Server is dual Xeon with 2G RAM, opsys is debian linux. Latest trunk red5 (but error is nearly the same with all previous I tried from 0.5).
Only 1 app running in red5, that serves 3 playlist from flv-s. Each filelist has about 100 flvs repeated about 1000 times (I can't find how to repeat the full playlist, sorry), each flv is 10-150M.
I limited the concurent user's number with my app to 100, because of the CPU limitation. I also disabled the connect of all clients while the application is initialising (read some database, and prepare the playlist).

public boolean connect(IConnection conn, IScope scope, Object[] params) {

if( initDone==false ) {

log.info( "Server is not started yet." );
rejectClient( "Server is not started yet." );

}
boolean isAdmin;

if( params==null params.length==0 ) {

isAdmin=false;

} else {

isAdmin=params[0].toString().equals( "1" );

}

if( isServerFull() && !isAdmin ) {

log.info( "Server is overloaded." );
rejectClient( "Server is overloaded." );

}
if (!super.connect(conn, scope, params)) {

return false;

}
return true;

}

With 80-100 user Red5 is leaking, about 2-3M/5sec.

1st try: I made a scheduled call of Runtime.getRuntime().gc() in my app in every 30sec. It helped, Red5 leaking, but GC collects the memory well. I had 1300-1400M free mem (Runtime.getRuntime().freeMemory()) when server reached 100 user. But after 1-2 hour Red5 start to leak hard, and gc can't release as much as memory the server eat, so it run out of memory.

2nd try: I put Runtime.getRuntime().runFinalization() before Runtime.getRuntime().gc() (I'm a noob in java, im flash programmer, just tried it)
Result is the same.

3rd try: I modified the scheduled routine, to check if the free memory is less than 500M, then close all client of the application. So I called this routine in my App:

private void closeAllClient() {

log.info( "Closing all clients:" );
Set<IClient> clients=appScope.getClients();
Iterator iClients = clients.iterator();
while( iClients.hasNext() ) {

IClient client=(IClient)iClients.next();
log.info( " " + client );
client.disconnect();

}

}

It worked fine, all application is closed (i put log.info into the appDisconnect function), but not much memory is released (10-20 megabytes, but not 900M, what is leaked till this time).

4. try: I modified the scheduled routine in my app to close all clients and restart my streams under 500M free mem.
I call this function, and the initChannel function (too long to copy-paste here):

private void closeAllChannel() {

log.info( "Closing all channels:" );
for( int i=0; i<channels.size(); i++ ) {

IServerStream channel=(IServerStream)channels.elementAt( i );
log.info( " " + channel );
channel.close();

}
channels=null;

}

The result is: Red5 halt with 900M free mem:-( Because of this the scheduled routine not called, the only I see in nohup.out that org.red5.server.BaseConnection.connect is works. The problem is, that I can't reach my application from BaseConnection to restart the streams, the app, or do something with this.

So in my case Red5 can hang up with 900M free mem too, but the most common hang up is to work fine for 2 hour, then eat all memory in 2 min, then OutOfMemory exception in FlowControlService or some other classes. But org.red5.server.BaseConnection.connect works after the streams, scheduling, and other application routines (appConnect, etc) hanged up.

[chameleon]

Your problem sounds exactly like the issue I am experiencing. My server also typically lasts around 2 hours before leaking all it's memory in a couple minute period. Thank you for your report, hopefully a couple different experiences with the problem may help point out where the problem lies.

[nancsi]

Yes, I found them similar, thats why I attached my comment to your report:-)
I made further testing on my server. I made app's "reboot" code static, and this way it can be called from anywhere in red5. I updated to the newest trunk, put my memory check routine into org.red5.server.BaseConnection.connect:
...

if( Runtime.getRuntime().freeMemory()<1024*1024*1024 ) {

reBoot();

}

...

private void reBoot() {

myApp.Application.reBoot();

}

(I imported myApp.Application in the begining of BaseConnection of course).
myApp.Application.reBoot() does the following:
- make connect unavailable for new clients
- disconnects all clients
- close all channels
- run garbage collector.
- make connect available for new clients
As red5 started to leak heavily this reboot routine called fine (freeMem in normal case is 1350-1450M, I set the leak-detect value to 1024M). It closed all the clients, flv playlist channels, recreated the channels, and run the garbage collector. When the process started there was 1007M free, when it finished there was 1148M free. So 200-300M mem is lost, nearly full restart of the application didn't help.
But the server worked after the restart, I can see streaming videos for some seconds min. Because then the freeMem run under 1024M, and the reboot is called again.
I made a trick: limited the max number of users to 30. The server's memory - with the scheduled, 30 sec full garbage collector - is grow up slowly to 1486M (normal value), but it leaked heavily, about 150M/30sec with 30 users, 250M/30sec with 40 users. If I increased the user's number the leak will be faster, if I decreased it, the leak will be slower, nearly linear, about 5M/user/30sec. Under 30 user the 30 sec scheduled garbage collector could handle the leaking.
So my discussion is that this memory leak is depends on the number of the stream viewers connected to the server, when it starts it eats about 10M/min/stream watcher client.

[nancsi]

Another try today:
I set the full gc-s routine interval very frequent, 5 sec (30 sec before).
In the begining, the free memory was nearly 1.5G after each gc run, tipical is:

[java] [INFO] 613144 DefaultQuartzScheduler_Worker-4:( ... ) Force garbage collector
[java] [INFO] 613145 DefaultQuartzScheduler_Worker-4:( ... ) memory free before: 1515095232 (1444M)
[java] [INFO] 613565 DefaultQuartzScheduler_Worker-4:( ... ) memory free after: 1563353664 (1490M)

After it fall down to 1G, and trhow an exception:

[java] [INFO] 5398849 pool-1-thread-3:( ... ) New client connected. Current viewers: 71 peek viewers: 100
[java] [INFO] 5398850 pool-1-thread-3:( ... ) memory free before: 1104027552 (1052M)
[java] [INFO] 5398851 pool-1-thread-3:( ... ) memory free after: 1103958120 (1052M)
[java] [ERROR] 5398855 DefaultQuartzScheduler_Worker-2:( org.red5.server.BaseConnection.error ) Error while disconnecting from scope null
[java] java.lang.NullPointerException
[java] at java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:157)
[java] at java.util.concurrent.ConcurrentHashMap.containsKey(ConcurrentHashMap.java:745)
[java] at org.red5.server.Scope.disconnect(Scope.java:328)
[java] at org.red5.server.BaseConnection.close(BaseConnection.java:176)
[java] at org.red5.server.net.rtmp.RTMPConnection.close(RTMPConnection.java:314)
[java] at org.red5.server.net.rtmp.RTMPMinaConnection.close(RTMPMinaConnection.java:92)
[java] at org.red5.server.net.rtmp.RTMPMinaConnection.onInactive(RTMPMinaConnection.java:97)
[java] at org.red5.server.net.rtmp.RTMPConnection$KeepAliveJob.execute(RTMPConnection.java:593)
[java] at org.red5.server.scheduling.QuartzSchedulingServiceJob.execute(QuartzSchedulingServiceJob.java:45)
[java] at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
[java] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)

Then another 8 of this type of exception.
Then finally:

[java] [INFO] 5610901 DefaultQuartzScheduler_Worker-2:( ... ) Force garbage collector
[java] [INFO] 5610902 DefaultQuartzScheduler_Worker-2:( ... ) memory free before: 1090867592 (1040M)
[java] [INFO] 5615633 DefaultQuartzScheduler_Worker-2:( ... ) memory free after: 1090936344 (1040M)
[java] [INFO] 5616136 pool-1-thread-12:( org.red5.server.BaseConnection.info ) BaseConnection.connect
[java] [INFO] 5616137 pool-1-thread-9:( org.red5.server.BaseConnection.info ) BaseConnection.connect
[java] Exception in thread "FlowControlService" java.lang.OutOfMemoryError: Java heap space

And the server halted completly (with 1G free mem).

[joachim]

The workaround for this is to disable the bandwidth control. See r1715 ( http://mirror1.cvsdude.com/trac/osflash/red5/changeset/1715) for details on how to do this. Assigning to Steven as he is currently working on a solution so the bw control can be enabled without any negative impact.

[steven]

Won't be fixed because of the new code of bandwidth controller since r1743

[Owens29Rita]

According to my own analysis, billions of persons in the world get the <a href=" http://bestfinance-blog.com">loans</a> at well known banks. Thus, there's a good possibility to get a student loan in every country.

[FerrellDelores27]

Do not guess that is easy to do research papers! There are lots of fraud custom research papers writing companies. That is not easy to find good firm.

[RoxannePace24]

Do you think that nobody is ready to help you with your research essays proofreading? That's not right, because research paper shop should assist you every time you request!

[AvilaMelinda19]

Don't you get what college guys seem to be aimed at? These guys people are concentrated on high academic grades! But, to receive supreme quality academic papers you need to have someone such as online term paper writing services to assist you.

[RondaLeonard]

Various persons have different future targets. Bright academic career seems to be the primary goal of a lot of guys and they can do everything to gain that. They can even order custom academic papers writing at essay writing service. Various methods are executable.

[Brittney32Burnett]

Sites' owners need the rss feed submission sites, when are willing the sites' contents to have greater perceptibility.

[Albert23JESSIE]

I know that rss feeds submission can give the best and most relevant traffic to your website and rss feed submissions service will help to do that.

[LYNNETTEHOLCOMB]

Cool little post! Thanks for taking the time to compile this. I also recognized more than 50% of ideas: If grinding this puzzle not an easy calling to confront with, I would buy Analytical Essay and get a top quality custom written paper performed by writing services’ writers. Great taste can't be denied!!!

[JewelWallace27]

If you try to find locality where you can buy term papers or buy research paper here is very complete place for you about essays writing, which afford examples and gives an fortune to learn how make audit . But this site is more charismatic, and more cooperative. So don't be lazy and write your own or buy essays about this good post. Thanks.