I've been looking into memory issues we have been having here for the last year on and off. While the problem occured maybe once a month before, it began to be more widespread in the last couple months.
We started by running ANTS Profiler to analize what memory our application was using, however the information returned was not terribly helpful due to the fact that testing was horribly slow while profiling the application.
Afterwards we downloaded DebugDiag from microsoft to see if we could take a memory dump and analize it with the memory report. The first crash dump we analized showed the following :
| Size of largest free VM block |
62,40 MBytes |
| Free memory fragmentation |
93,59% |
| Free Memory |
972,84 MBytes (47,50% of Total Memory) |
| Reserved Memory |
230,84 MBytes (11,27% of Total Memory) |
| Committed Memory |
844,26 MBytes (41,22% of Total Memory) |
| Total Memory |
2,00 GBytes |
| Largest free block at |
0x00000000`5376a000 |
The fragmentation was incredibly high. What was impressive is that it only took a about 20 minutes to get this high level of memory fragmentation. Fragmentation typically occurs when an object which was collected by garbage collection is marked as a "free" block. which means it's ready to be reclaimed, however the object occupying the next address space is pinned. To find out what the cause of this is, we had to revert to WinDbg.
WinDbg allows us to see alot of information regarding what is happening internally, also in a "live" environment. Additionally it gives us the ability to set breakpoints and really debug the application without having visual studio and almost without any performance hit like we hit with ANTs.
To set WinDbg up on a server, you need to download Debugging Tools for Windows. The next thing you need is to set the symbol path under the file menu and use something similar to :
srv*\\fileserver\development\debuggingsymbols*http://msdl.microsoft.com/download/symbols
This will set windbg up to search for pdb files within the file://fileserver/development/debuggingsymbols (or wherever you wish), and those not found which are searched for from microsofts symbol server.
After doing this, you can attach to a process (in my case a iis worker process) with F6. After which you will come in to the console, here you want to type :
.load clr10/sos.dll
Which loads the son of strike dll for managed debugging (this is mainly for .NET 1.1 debugging). From there, you are ready to go. You can see the commands available by typing "!help"
We knew we had fragmentation, so we started with
!dumpheap -stat
0:000> !dumpheap -stat
Loaded Son of Strike data table version 5 from "C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll"
Loading the heap objects into our cache.
------------------------------
Heap 0
total 3,078,969 objects
------------------------------
Heap 1
total 2,438,004 objects
------------------------------
Heap 2
total 2,542,493 objects
------------------------------
Heap 3
total 3,619,032 objects
------------------------------
total 11,678,498 objects
Statistics:
MT Count TotalSize Class Name
0x79c07d54 1 12 System.Text.Encoding/DefaultEncoder
0x79c07480 1 12 System.LdsSyncHelper
0x79bf9ab0 1 12 System.Threading.CompressedStack
0x79bc185c 1 12 System.Int16
0x79bc0970 1 12 System.Empty
...
0x029342ac 1,750,261 35,005,220 System.Collections.Specialized.ListDictionary/DictionaryNode
0x000db8e8 108 35,009,756 Free
0x02d6584c 514,569 39,107,244 System.Web.UI.LiteralControl
0x02d6751c 617,033 49,362,640 System.Web.UI.HtmlControls.HtmlTableCell
0x01ea209c 819,948 53,246,984 System.Object[]
0x79b946b0 1,365,250 321,240,224 System.String
Total 11,678,498 objects, Total size: 687,433,004
Fragmented blocks larger than 0.5MB:
Addr Size Followed by
0x2770e3dc 4.6MB large free object followed by 0x27bae03c System.Net.Connection
this shows all objects and the count/size of the objects in memory. From there we found that the Free object had a method table of 000db8e8. To see all instances of this method table, we used "!dumpheap -mt 000db8e8" where we received the following.
0:019> !dumpheap -mt 000db8e8
Using our cache to search the heap.
Address MT Size Gen
0x10030030 0xdb8e8 12 2 Free
0x1003003c 0xdb8e8 12 2 Free
0x100fcb7c 0xdb8e8 316 2 Free
0x344d2af4 0xdb8e8 12 2 Free
0x364f1cc8 0xdb8e8 12 1 Free
0x20030840 0xdb8e8 1,984 3 Free
0x20032810 0xdb8e8 2,032 3 Free
0x20034810 0xdb8e8 2,032 3 Free
0x20035810 0xdb8e8 16 3 Free
0x20036820 0xdb8e8 16 3 Free
0x20037040 0xdb8e8 16 3 Free
0x20037860 0xdb8e8 16 3 Free
0x20038080 0xdb8e8 16 3 Free
0x2003b0b0 0xdb8e8 4,096 3 Free
0x2003e0b0 0xdb8e8 4,096 3 Free
0x2003f0b0 0xdb8e8 96,296 3 Free
0x200578d8 0xdb8e8 131,176 3 Free
0x20078940 0xdb8e8 4,096 3 Free
0x2007a940 0xdb8e8 4,096 3 Free
0x2007c940 0xdb8e8 4,096 3 Free
0x2007e940 0xdb8e8 4,096 3 Free
0x20080940 0xdb8e8 62,888 3 Free
0x20090ee8 0xdb8e8 4,096 3 Free
0x20092ee8 0xdb8e8 4,096 3 Free
0x200946f8 0xdb8e8 6,128 3 Free
0x20096ee8 0xdb8e8 4,096 3 Free
0x20098ee8 0xdb8e8 4,096 3 Free
0x2009aee8 0xdb8e8 92,736 3 Free
0x200b2138 0xdb8e8 6,128 3 Free
0x200b4138 0xdb8e8 131,112 3 Free
0x200d5160 0xdb8e8 4,096 3 Free
0x200d6970 0xdb8e8 6,128 3 Free
0x200d9160 0xdb8e8 241,752 3 Free
0x201151b8 0xdb8e8 4,096 3 Free
0x201179c8 0xdb8e8 2,032 3 Free
0x201191b8 0xdb8e8 4,096 3 Free
0x2011b1b8 0xdb8e8 4,096 3 Free
0x2011d1b8 0xdb8e8 4,096 3 Free
0x2011f1b8 0xdb8e8 177,320 3 Free
0x2014ae70 0xdb8e8 400 3 Free
0x2014c000 0xdb8e8 4,096 3 Free
0x2014d810 0xdb8e8 129,048 3 Free
0x2016d838 0xdb8e8 418,752 3 Free
0x201d4408 0xdb8e8 52,752 3 Free
0x201e2218 0xdb8e8 832,576 3 Free
0x202ade68 0xdb8e8 142,272 3 Free
0x202d1a28 0xdb8e8 131,112 3 Free
0x202f2260 0xdb8e8 135,448 3 Free
0x20313b88 0xdb8e8 521,136 3 Free
0x20393748 0xdb8e8 6,128 3 Free
0x20395f38 0xdb8e8 4,096 3 Free
0x20397f38 0xdb8e8 857,296 3 Free
0x2046a408 0xdb8e8 4,096 3 Free
0x2046c408 0xdb8e8 4,096 3 Free
0x2046e408 0xdb8e8 4,096 3 Free
0x20470408 0xdb8e8 4,096 3 Free
0x20472408 0xdb8e8 4,096 3 Free
0x20474408 0xdb8e8 4,096 3 Free
0x20476408 0xdb8e8 4,096 3 Free
0x20478408 0xdb8e8 4,096 3 Free
0x2047a408 0xdb8e8 251,224 3 Free
0x204b8960 0xdb8e8 4,096 3 Free
0x204ba960 0xdb8e8 4,096 3 Free
0x204bc170 0xdb8e8 6,128 3 Free
0x204be960 0xdb8e8 4,096 3 Free
0x204c0960 0xdb8e8 917,728 3 Free
0x205a1a40 0xdb8e8 87,560 3 Free
0x205b8048 0xdb8e8 1,767,352 3 Free
0x207ca1e0 0xdb8e8 16 3 Free
0x26030030 0xdb8e8 1,721,728 3 Free
0x26236f90 0xdb8e8 16 3 Free
0x262fc358 0xdb8e8 524,320 3 Free
0x263fc388 0xdb8e8 16 3 Free
0x2647c3a8 0xdb8e8 16 3 Free
0x264fc3c8 0xdb8e8 16 3 Free
0x2657c3e8 0xdb8e8 16 3 Free
0x265fc408 0xdb8e8 16 3 Free
0x2667c428 0xdb8e8 16 3 Free
0x266fc448 0xdb8e8 752,048 3 Free
0x267fecb8 0xdb8e8 16 3 Free
0x14030030 0xdb8e8 12 2 Free
0x1403003c 0xdb8e8 12 2 Free
0x149c70d8 0xdb8e8 680 2 Free
0x6ad4bf54 0xdb8e8 12 2 Free
0x6c5bffb8 0xdb8e8 12 1 Free
0x6c5bffc4 0xdb8e8 12 0 Free
0x18030030 0xdb8e8 12 2 Free
0x1803003c 0xdb8e8 12 2 Free
0x18c28630 0xdb8e8 476 2 Free
0x18d0c5d4 0xdb8e8 1,068 2 Free
0x2770e3dc 0xdb8e8 4,848,736 2 Free
0x3897b358 0xdb8e8 12 1 Free
0x3897b364 0xdb8e8 12 0 Free
0x22190098 0xdb8e8 16 3 Free
0x221a72f0 0xdb8e8 16 3 Free
0x221a7300 0xdb8e8 9,251,576 3 Free
0x1c030030 0xdb8e8 12 2 Free
0x1c03003c 0xdb8e8 12 2 Free
0x1c22f0dc 0xdb8e8 21,656 2 Free
0x1d727e30 0xdb8e8 2,640 2 Free
0x3e08acb8 0xdb8e8 140 2 Free
0x30ed9410 0xdb8e8 12 2 Free
0x33adaec8 0xdb8e8 12 1 Free
0x23050088 0xdb8e8 16 3 Free
0x23067220 0xdb8e8 16 3 Free
0x2307e478 0xdb8e8 16 3 Free
0x2307e488 0xdb8e8 8,133,072 3 Free
0x6f20030 0xdb8e8 2,419,448 3 Free
Statistics:
MT Count TotalSize Class Name
0x000db8e8 108 35,009,756 Free
Total 108 objects, Total size: 35,009,756
The blue area above stood out as due to the fact that these addresses were within a single GC Heap in the large object heap as below by using "!eeheap -gc".
0:019> !eeheap -gc
Number of GC Heaps: 4
------------------------------
Heap 0 (0x000dbb68)
generation 0 starts at 0x364f1cd4
generation 1 starts at 0x364f1cc8
generation 2 starts at 0x10030030
ephemeral segment allocation context: none
segment begin allocated size reserved
0x10030000 0x10030030 0x1402ff50 0x03ffff20(67,108,640) 0x00090000
0x2c030000 0x2c030030 0x3002ffc4 0x03ffff94(67,108,756) 0x04000000
0x34030000 0x34030030 0x364f1ce0 0x024c1cb0(38,542,512) 0x012d2000
Large object heap starts at 0x20030030
segment begin allocated size reserved
0x20030000 0x20030030 0x2088f5a8 0x0085f578(8,779,128) 0x00780000
0x26030000 0x26030030 0x26849b88 0x00819b58(8,493,912) 0x007c6000
Heap Size 0xb53ac34(190,032,948)
------------------------------
Heap 1 (0x000dc288)
generation 0 starts at 0x6c5bffc4
generation 1 starts at 0x6c5bffb8
generation 2 starts at 0x14030030
ephemeral segment allocation context: none
segment begin allocated size reserved
0x14030000 0x14030030 0x1802ff60 0x03ffff30(67,108,656) 0x00090000
0x685c0000 0x685c0030 0x6c5bffd0 0x03ffffa0(67,108,768) 0x00b30000
Large object heap starts at 0x0b820030
segment begin allocated size reserved
0x0b820000 0x0b820030 0x0b820030 0x00000000(0) 0x00fdf000
Heap Size 0x7ffff00(134,217,472)
------------------------------
Heap 2 (0x000dcd98)
generation 0 starts at 0x3897b364
generation 1 starts at 0x3897b358
generation 2 starts at 0x18030030
ephemeral segment allocation context: none
segment begin allocated size reserved
0x18030000 0x18030030 0x1bf9f380 0x03f6f350(66,515,792) 0x00090000
0x27030000 0x27030030 0x2b02ffc0 0x03ffff90(67,108,752) 0x01000000
0x38030000 0x38030030 0x3897b370 0x0094b340(9,745,216) 0x01fbf000
Large object heap starts at 0x22030030
segment begin allocated size reserved
0x22030000 0x22030030 0x22af0c88 0x00ac0c58(11,275,352) 0x0051f000
Heap Size 0x937d278(154,653,304)
------------------------------
Heap 3 (0x000dd880)
generation 0 starts at 0x33adaed4
generation 1 starts at 0x33adaec8
generation 2 starts at 0x1c030030
ephemeral segment allocation context: none
segment begin allocated size reserved
0x1c030000 0x1c030030 0x2002fe08 0x03fffdd8(67,108,312) 0x00780000
0x3d030000 0x3d030030 0x4102ff60 0x03ffff30(67,108,656) 0x003cb000
0x30030000 0x30030030 0x33adaee0 0x03aaaeb0(61,517,488) 0x02d2e000
Large object heap starts at 0x23030030
segment begin allocated size reserved
0x23030000 0x23030030 0x2388f680 0x0085f650(8,779,344) 0x00780000
0x06f20000 0x06f20030 0x072f65b8 0x003d6588(4,023,688) 0x00c09000
Heap Size 0xc6e0790(208,537,488)
------------------------------
GC Heap Size 0x28f9853c(687,441,212)
To find out what was keeping these free objects from being released, we had to look at the objects after them.
0:019> !dumpobj 0x2003b0b0
Free Object
Size 4096(0x1000) bytes
0:019> !dumpobj 0x2003b0b0+1000
Name: System.Object[]
MethodTable 0x01ea209c
EEClass 0x01ea2018
Size 4096(0x1000) bytes
GC Generation: 3
Array: Rank 1, Type CLASS
Element Type: System.Object
Content: 1,020 items
This shows us that there is an object array following this free object. To find out what has pinned or has a handle on this object, we use gcroot as follows.
0:019> !gcroot 0x2003b0b0+1000
Scan Thread 13 (0x874)
Scan Thread 19 (0x228)
Scan Thread 20 (0xcb8)
Scan Thread 21 (0xb54)
Scan Thread 24 (0x8b8)
Scan Thread 25 (0x974)
Scan Thread 7 (0x864)
Scan Thread 6 (0xbc)
Scan Thread 26 (0xf80)
Scan Thread 22 (0xc10)
Scan Thread 31 (0x3cc)
Scan Thread 5 (0x200)
Scan Thread 32 (0x7ac)
Scan Thread 33 (0xa7c)
Scan Thread 34 (0x8c0)
Scan Thread 35 (0xa8c)
Scan Thread 36 (0x564)
Scan Thread 37 (0x7d8)
Scan Thread 38 (0x8ec)
Scan Thread 39 (0x140)
Scan Thread 40 (0xf68)
Scan Thread 41 (0x438)
Scan Thread 4 (0x770)
Scan HandleTable 0xd1010
Scan HandleTable 0xd88a8
Scan HandleTable 0x146030
Scan HandleTable 0x2673640
HANDLE(Strong):30a3020:Root:0x2003c0b0(System.Object[])->0x10af86cc(System.Net.Sockets.Socket)
This tells us that the object array is held by a socket, then we attempt to see what endpoint the socket has.
0:019> !dumpobj 0x10af86cc
Name: System.Net.Sockets.Socket
MethodTable 0x02ace4a4
EEClass 0x02ae039c
Size 64(0x40) bytes
GC Generation: 2
mdToken: 0x020002b8 (c:\windows\assembly\gac\system\1.0.5000.0__b77a5c561934e089\system.dll)
FieldDesc*: 0x02acd9f8
MT Field Offset Type Attr Value Name
0x02ace4a4 0x400143f 0x34 System.Boolean instance 0 incallback
0x02ace4a4 0x4001440 0x4 CLASS instance 0x00000000 m_AcceptQueue
0x02ace4a4 0x4001441 0x1c System.Int32 instance 3708 m_Handle
0x02ace4a4 0x4001442 0x8 CLASS instance 0x10af888c m_RightEndPoint
0x02ace4a4 0x4001443 0xc CLASS instance 0x00000000 m_LocalEndPoint
0x02ace4a4 0x4001444 0x10 CLASS instance 0x00000000 m_RemoteEndPoint
0x02ace4a4 0x4001445 0x35 System.Boolean instance 1 m_WasConnected
0x02ace4a4 0x4001446 0x36 System.Boolean instance 0 m_WasDisconnected
0x02ace4a4 0x4001447 0x37 System.Boolean instance 1 willBlock
0x02ace4a4 0x4001448 0x38 System.Boolean instance 1 willBlockInternal
0x02ace4a4 0x4001449 0x20 System.Int32 instance 2 addressFamily
0x02ace4a4 0x400144a 0x24 System.Int32 instance 1 socketType
0x02ace4a4 0x400144b 0x28 System.Int32 instance 6 protocolType
0x02ace4a4 0x400144c 0x39 System.Boolean instance 1 m_Bound
0x02ace4a4 0x400144d 0x14 CLASS instance 0x00000000 m_AsyncEvent
0x02ace4a4 0x400144e 0x2c System.Int32 instance 0 m_BlockEventBits
0x02ace4a4 0x400144f 0x18 CLASS instance 0x10af889c m_PermittedRemoteAddress
0x02ace4a4 0x4001451 0x30 System.Int32 instance 0 m_IntCleanedUp
0x02ace4a4 0x400143b 0x4 System.Boolean shared static m_Initialized
>> Domain:Value 0x000d66e0:NotInit 0x001036c0:1 0x0017c360:1 <<
0x02ace4a4 0x400143c 0x8 System.Boolean shared static m_SupportsIPv4
>> Domain:Value 0x000d66e0:NotInit 0x001036c0:1 0x0017c360:1 <<
0x02ace4a4 0x400143d 0xc System.Boolean shared static m_SupportsIPv6
>> Domain:Value 0x000d66e0:NotInit 0x001036c0:0 0x0017c360:0 <<
0x02ace4a4 0x400143e 0 CLASS shared static m_SyncObject
>> Domain:Value 0x000d66e0:NotInit 0x001036c0:0x14079654 0x0017c360:0x181b3c60 <<
0x02ace4a4 0x4001450 0x10 System.Boolean shared static UseOverlappedIO
>> Domain:Value 0x000d66e0:NotInit 0x001036c0:0 0x0017c360:0 <<
0:019> !gcroot 0x10af86cc
Scan Thread 13 (0x874)
Scan Thread 19 (0x228)
Scan Thread 20 (0xcb8)
Scan Thread 21 (0xb54)
Scan Thread 24 (0x8b8)
Scan Thread 25 (0x974)
Scan Thread 7 (0x864)
Scan Thread 6 (0xbc)
Scan Thread 26 (0xf80)
Scan Thread 22 (0xc10)
Scan Thread 31 (0x3cc)
Scan Thread 5 (0x200)
Scan Thread 32 (0x7ac)
Scan Thread 33 (0xa7c)
Scan Thread 34 (0x8c0)
Scan Thread 35 (0xa8c)
Scan Thread 36 (0x564)
Scan Thread 37 (0x7d8)
Scan Thread 38 (0x8ec)
Scan Thread 39 (0x140)
Scan Thread 40 (0xf68)
Scan Thread 41 (0x438)
Scan Thread 4 (0x770)
Scan HandleTable 0xd1010
Scan HandleTable 0xd88a8
Scan HandleTable 0x146030
HANDLE(Strong):2593a70:Root:0x38924e60(System.Net.Sockets.OverlappedAsyncResult)->0x10af86cc(System.Net.Sockets.Socket)
Scan HandleTable 0x2673640
0:019> !dumpobj 0x10af888c
Name: System.Net.IPEndPoint
MethodTable 0x02b305e4
EEClass 0x02ae0d34
Size 16(0x10) bytes
GC Generation: 2
mdToken: 0x02000242 (c:\windows\assembly\gac\system\1.0.5000.0__b77a5c561934e089\system.dll)
FieldDesc*: 0x02b304c4
MT Field Offset Type Attr Value Name
0x02b305e4 0x4001134 0x4 CLASS instance 0x10af8744 m_Address
0x02b305e4 0x4001135 0x8 System.Int32 instance 80 m_Port
0x02b305e4 0x4001136 0 CLASS shared static Any
>> Domain:Value 0x000d66e0:NotInit 0x001036c0:0x14077a14 0x0017c360:NotInit <<
0x02b305e4 0x4001137 0x4 CLASS shared static IPv6Any
>> Domain:Value 0x000d66e0:NotInit 0x001036c0:0x14077a40 0x0017c360:NotInit <<
0:019> !PrintIPAddress 0x10af8744
127.0.0.1
This shows that the culprit is an open connection to port 80 on the local machine, which was the result of a webservice call (where the hostname used was within hosts and pointing to 127.0.0.1). Within code we found that web services for the application in question were static and were used a great deal during a single rendering of a page for referencing. The underlying connections for these services remain in memory and are the guilty party, our immediate solution was to make them threadstatic to allow the webservice object to die when the thread is collected by garbage collection. The proper solution was to encapsulate the webservice with a using statement, but due to the nature of the application this wasnt possible in any easy fasion, so we opted for a halfway solution until we can fully correct the issue.
This was my first WinDbg experience. Amazingly powerfull tool and it is a huge help when supporting applications and stop the guesswork when addressing the really hard problems when developing or supporting .NET applications.
Subscribe