Advanced Memory Allocation 内存分配进阶
Advanced Memory Allocation 內存分配進階
URL: ?http://blog.csdn.net/amwihihc/article/details/7481656
May 01, 2003 By Gianluca Insolvibile in Embedded Software
Call some useful fuctions of the GNU C library to save precious memory and to find nasty bugs.
Dealing with dynamic memory traditionally has been one of the most awkward issues of C and C++ programming. It is not surprising that some supposedly easier languages, such as Java, have introduced garbage collection mechanisms that relieve programmers of this burden. But for hard-core C programmers, the GNU C library contains some tools that allow them to tune, check and track the usage of memory.
處理動態內存在c/c++編程中,一直被認為是一件很難操作的事情.在一些被認為是比較簡單的編程語言,像Java,引入垃圾回收機制來減輕程序員的負擔,也就不會讓人覺得驚訝了.但是對一些c程序員來說,GNU C庫包含一些工具,能夠允許程序員調整,檢查,以及跟蹤內存的使用.
Memory Management Basics
內存管理基本知識
A process' memory usually is classified as either static, the size is predetermined at compile time, or dynamic, space is allocated as needed at runtime. The latter, in turn, is divided into heap space, where malloc()'d memory comes from, and stack, where functions' temporary work space is placed. As Figure 1 shows, heap space grows upward, whereas stack space grows downward.
一個進程的內存,通常被分為靜態和動態兩類.靜態內存的大小在編譯期就被已經確定了,而動態內存卻是在運行時按需分配的.后者還可以被分為堆空間和棧空間,堆空間是用malloc分配得到的,而函數的臨時工作空間被安置在棧上.如圖1所示,堆空間向上增長,而棧空間向下增長.
Figure 1. The heap and stack grow toward each other.
圖1.堆和棧相向增長
When a process needs memory, some room is created by moving the upper bound of the heap forward, using the brk() or sbrk() system calls. Because a system call is expensive in terms of CPU usage, a better strategy is to call brk() to grab a large chunk of memory and then split it as needed to get smaller chunks. This is exactly what malloc() does. It aggregates a lot of smaller malloc() requests into fewer large brk() calls. Doing so yields a significant performance improvement. The malloc() call itself is much less expensive than brk(), because it is a library call, not a system call. Symmetric behavior is adopted when memory is freed by the process. Memory blocks are not immediately returned to the system, which would require a new brk() call with a negative argument. Instead, the C library aggregates them until a sufficiently large, contiguous chunk can be freed at once.
當一個進程需要內存時,通過brk()或sbrk()系統調用,移動堆的上邊界,產生一些空間.因為一次系統調用是一個非常消耗cpu的操作,所以,一個比較好的策略就是,調用brk()來獲取一大區塊內存,然后按需切割他們來得到較小的內存區塊.這就是malloc()函數的所作的事情.它將大量的小區塊malloc操作聚集到較少的大塊的brk()操作.這樣做產生了顯著的性能改善.malloc調用它自己比調用brk()的代價要小的多.因為前者是一個庫調用,而不是一個系統調用.對稱的動作也被使用在內存釋放的時候.內存塊并不立即返回給系統,如果要立即返回給系統,將需要調用brk(),并給它一個負的參數.相反,c庫把這些內存塊收集起來,直到足夠的大,連續的內存區塊可以一次全部釋放.
For very large requests, malloc() uses the mmap() system call to find addressable memory space. This process helps reduce the negative effects of memory fragmentation when large blocks of memory are freed but locked by smaller, more recently allocated blocks lying between them and the end of the allocated space. In this case, in fact, had the block been allocated with brk(), it would have remained unusable by the system even if the process freed it.
對一些非常大的請求,malloc()使用mmap()系統調用來尋找可尋址的內存空間.這個處理方法幫助減少內存碎片的負面影響,內存碎片的產生是由于一大塊內存被free,但是仍被一些小的,最近分配的,躺在這些大內存塊和分配空間結束位置之間的這些小內存塊鎖定,在這種情況下,事實上,假設是使用brk()分配的,對系統來說不能使用,即使進程已經釋放了它.
Library functions that deal with dynamic memory are not limited to malloc() and free(), although these are by far the most-used calls. Other available functions include realloc(), to resize an already allocated block; calloc(), to allocate a cleared block; and memalign(), posix_memalign() and valloc(), to allocate an aligned block.
庫函數處理動態內存并不局限于malloc()和free(),盡管至今為止它們是最常使用的調用.其它一些可用的函數包括realloc(),可以用來重置已分配內存塊的大小,calloc(),來分配一個被清0的內存塊,以及memalign(),posix_align(),分配一個對齊內存塊. Dealing with Memory Status 處理內存狀態
The strategy adopted by the C library memory management code is optimized for generic memory usage profiles. Although this strategy produces good performance in most cases, some programs might benefit from slightly different parameter tuning. First, check your memory usage statistics by using either the malloc_stats() or the mallinfo() library calls. The former prints as a standard error a brief summary of memory usage in the program. This summary includes how many bytes have been allocated from the system, gathered with brk(); how many are actually in use, found with malloc(); and how much memory has been claimed, using mmap(). Here is a sample output:
c庫內存管理所采用的策略,為適合通用的內存使用情況而作了優化.盡管這個策略在大多數情況下性能表現良好,一些程序可能會從一些稍微不同函數調整中受益.首先,檢查你的內存使用統計,通過使用malloc_stats()或是mallinfo()系統調用.前者以一個標準錯誤的方式,在程序中打印內存使用概要.這個概要包括通過brk(),多少字節從系統中分配出去;實際通過malloc使用的多少;以及使用mmap(),多少內存被聲明使用.以面是一個簡單的輸出:
If you need to have more precise information and want to make more than a printout, mallinfo() is helpful. This function returns a struct mallinfo containing various memory-related status indicators; the most interesting are summarized in the Sidebar “Useful Parameters Provided by mallinfo”. For a complete description of the structure, take a look at /usr/include/malloc.h.
如果你還需要更精確的信息,并且想要更多的輸入,mallinfo()就非常有用.這個函數返回一個maillinfo結構體,包含各種內存相關的狀態指示器,最有趣的東西被總結在"Useful Parameters Provided by mallinfo".對這個結構休的完整描述,參見/usr/include/malloc.h.
Useful Parameters Provided by mallinfo()
mallinfo提供的有用的參數
Another useful function provided by libc is malloc_usable_size(), which returns the number of bytes you actually can use in a previously allocated memory block. This value may be more than the amount you originally requested, due to alignment and minimum size constraints. For example, if you allocate 30 bytes, the usable size is actually 36. This means you could write up to 36 bytes to that memory block without overwriting other blocks. This is an extremely awful and version-dependent programming practice, however, so please don't do it. The most useful application of malloc_usable_size() probably is as a debug tool. For example, it can check the size of a memory block passed from outside before writing to it.
另一個由libc提供的用的函數是malloc_usable_size(),它返回在一個預先分配的內存塊里你實際能使用的字節數。這個值可能會比你最初請求的值要大,因為內存齊和最小內存分配值約束。例如,如果你分配30字節,但是可使用的的大小是36,這意味著你可以向那塊內存寫入36個字節而不會覆蓋其它內存塊。這是一個非常糟糕和依賴版本的編程實踐,然而,請不要這要做。malloc_usable_size()最有用的使用可能是作為一個調試工具。例如,它能夠在寫入一個從外部傳入的內存塊之前,檢查它的大小。 Controlling the Allocation Strategy 分配策略控制
You can alter the behavior of the memory management functions by adjusting some of the parameters exposed by the mallopt() function (Listings 1 and 2).
你可以定制內存管理函數的行為,通過調整由mallopt()函數的參數。
Listing 1. Setting the Trim Threshold with mallopt()
使用mallopt()設置內存消減的閾值。
Listing 2. A smaller trim threshold might save space.
一個更小的內存消減閾值可能節省內存空間。
The prototype of this function and a basic set of four parameters are part of the SVID/XPG/ANSI standard. The current GNU C library implementation (version 2.3.1 as of this writing) honors only one of them (M_MXFAST), leaving three out. On the other hand, the library provides four additional parameters not specified by the standard. Tunable parameters accepted by mallopt() are described in the Sidebar “Tunable Parameters for mallopt()”.
這個函數的原型以及四個參數的簡單集合是SVID/XPG/ANSI標準的一部分。當前的GNU C庫的實現(寫本文時,版本號為2.3.1)只支持它們中的一個(M_MXFAST),而不支持其它三個。另一方面,這個庫提供四個額外的參數,而不是通過標準指定。mallopt()接受的可調整的參數在下一章中有描述。
Tunable Paramenter for mallopt()
mallopt()可調整的參數
Allocation tuning is possible even without introducing mallopt() calls inside your program and recompiling it. This may be useful if you want to test values quickly or if you don't have the sources. All you have to do is set the appropriate environment variable before running the application. Table 1 shows the mapping between mallopt() parameters and environment variables, as well as some additional information. If you wish to set the trim threshold to 64KB, for example, you can run this program:
內存分配調整甚至可以不在你的程序中引入mallopt()調用和重新編譯它。在你想快速測試一些值或者你沒有源代碼時,這非常有用。你僅需要做的是在運行程序前,設置合適的環境變量。表1展示mallopt()參數和環境變量的映射關系以及一些額外的信息。例如,如果你希望設置內存消減閾值為64k,你可以運行這個程序:
MALLOC_TRIM_THRESHOLD=65536 my_prog
Speaking of trimming, it is possible to trim the memory arena and give any unused memory back to the system by calling malloc_trim(pad). This function resizes the data segment, leaving at least pad bytes at the end of it and failing if less than one page worth of bytes can be freed. Segment size is always a multiple of one page, which is 4,096 bytes on i386. The size of the memory available to be trimmed is stored in the keepcost parameter of the struct returned by mallinfo(). Automatic trimming is done inside the free() function by calling memory_trim(), if the current value of keepcost is higher than the M_TRIM_THRESHOLD value, and by using the value of M_TOP_PAD as the argument.
說到內存消減,它可以通過調用malloc_trim(pad)消減內存區域,將任何未使用的內存返回給系統。這個函數重新設置了數據段的大小,在數據段尾部保留至少pad字節,并且當少于一個頁大小的字節能夠被釋放時,將產生調用失敗。段大小總是頁大小的倍數,在i386上,頁大小是4096字節。能夠被消減的內存大小被保存在由mallinfo()返回的結構體的keepcost 字段上。如果當前keepcost的值比M_TRIM_THRESHOLD的值要大,那么在free()函數里通過調用memory_trim(),使用M_TOP_PAD的值作為參數,自動內存消減就被完成了。
Table 1. mallopt() Parameters Mapped to Environment Variables
| mallopt() option | Env var | Default value | Notes |
| M_TRIM_THRESHOLD | MALLOC_TRIM_THRESHOLD_ | 128KB | -1U disables |
| M_TOP_PAD | MALLOC_TOP_PAD_ | 0 | ? |
| M_MMAP_THRESHOLD | MALLOC_MMAP_THRESHOLD_ | 128KB | 0 disables |
| M_MMAP_MAX | MALLOC_MMAP_MAX_ | 64 |
Memory Debugging: Consistency Checks
內存調試:連續性檢查
Debugging memory is often one of the most time-consuming tasks when developing complex programs. The two basic aspects of this problem are checking memory corruption and tracing block allocation and release.
當開發復雜程序時,調試內存經常是一項非常耗時的任務。兩個基本角度是檢查內存越界和跟蹤內存塊分配和釋放。
Memory corruption happens when writing to a location lying inside the legal data segment but outside the boundaries of the memory block you intended to use. An example is writing beyond an array's end. In fact, if you were to write outside the legal data segment, a segmentation fault would halt the program immediately or trigger the appropriate signal handler, allowing you to identify the misbehaving instruction. Memory corruption is thus more subtle, because it can pass unnoticed and cause a faulty behavior in a part of the program quite far from the offending part. For this reason, the sooner you detect it in the program, the higher your chances are of catching the bug.
內存破壞發生在,當向有效數據段內的一個內存位置寫數據卻寫到了你意圖使用的內存塊的邊界之外。一個例子就是寫數據到數據尾部之后。事實上,如果你試圖寫數據到合法數據段之外,一個段錯誤將會立即中止你的程序,或者觸發一個適合的信號處理,它允許你鑒別錯誤行為的指令。內存破壞也難以捉摸,因為它能夠沒有任何提示的通過導致內存越界的代碼,并且在距離導致內存越界非常遠的程序部分造成一個段錯誤的行為。
Corruption may affect other memory blocks (messing with the application data) and the heap management structures. In the former case, the only symptom that something is going wrong comes from analyzing your own data structures. In the latter case, you can rely on some specific GNU libc consistency check mechanisms that alert you when something wrong is detected.
Memory checking in a program can be enabled as automatic or manual. The former is done by setting the environment variable MALLOC_CHECK_:
內存破壞可能影響其它的內存塊(擾亂程序數據)和堆管理結構。在前一個例子中,某些地方會出錯的惟一癥兆就是來自于分析你自己的數據結構。在接下來的例子中,你可以依賴于某些特定的GNU libc 連續性檢查機制,當某些地方的錯誤被檢測到的時候,它會發出警報。在程序中的內存檢查能夠通過手動或自動的啟動。前一種情況可以通過設置環境變量來完成:
MALLOC_CHECK_=1 my_prog
This mechanism is able to catch a fair number of boundary overflows and, in some cases, to protect the program from crashing. The action undertaken when a fault is detected depends on the value of MALLOC_CHECK_: 1 prints a warning message to stderr but does not abort the program; 2 aborts the program without any output; and 3 combines the effects of 1 and 2.
這個機制能夠捕獲大多數的邊界溢出并且,在某些情況下,可以避免程序崩潰。當一個錯誤被探測到的時候,這個機制所采取的行為依賴于MALLOC_CHECK的值,值為1時,打印一個警告信息到標準錯誤輸出但是不會終止程序,值為2時,終止程序,沒有任何輸出,值為3時,是1和2效果的聯合(打印警告信息,終止程序)。
Automatic checking takes place only when memory-related functions are invoked. That is, if you write beyond an array's end, it won't be noticed until the next malloc() or free() call. Also, not all the errors are caught, and the information you obtain is not always extremely useful. In the case of free(), you know which pointer was being freed when the error was detected, but that gives no hint whatsoever as to who trashed the heap. In the case of errors detected during an allocation, you merely receive a “heap corrupted” message.
只有當內存相關的函數被調用時,自動內存檢查才會發生。也就是說,如果你向一個數據的尾部之后寫數據,程序不會注意到,直到下個malloc()或者free()調用。同時,不是所有的錯誤被捕獲,并且你得到的信息也不總是非常有用。在free()的情況下,當錯誤被探測到的時候,你知道哪個指針被釋放掉,但是,那并沒有給那些搞亂堆的人任何提示。在分配期間內存,探測到錯誤的情況下,你僅僅收到一個“堆被破壞”的消息。
The alternative is to place manual checkpoints here and there in the program. To do this, you must call the mcheck() function at the beginning of the program. This function allows you to install a custom memory fault handler that can be invoked each time heap corruption is detected. A default handler also is available if you don't provide your own. Once mcheck() has been called, all the consistency checks you get with MALLOC_CHECK_ are in place. Moreover, you can call the mprobe() function manually to force a check on a given memory pointer at any time. Values returned by mprobe() are summarized in the Sidebar “mprobe() Results”.
可以考慮的解決方法是放置手動檢查點在程序的各個地方。為了這樣做,你必須在程序的開始處調用mcheck()函數。這個函數允許你安裝一個自定義的內存錯誤處理器,它能夠在每次堆破壞被探測到的時候被調用。如果你不提供你自己的,也可以使用默認的處理器。一旦mcheck()被調用,你通過設置MALLOC_CHECK_得到的連續性檢查就會生效。而且,你可以在任何時候調用手動的調用mprobe()函數來強制檢查一個給定的內存指針。mprobe()的返回值被總結在下一章。
mprobe() Results
mprobe() 結果
If you want to check the whole heap and not only one block, you can call mcheck_check_all() to walk through all the active blocks. You also can instruct the memory management routines to use mcheck_check_all(), instead of checking only the current block by initializing mcheck_pedantic() instead of mcheck(). Be aware, though, that this approach is rather time consuming.A third way to enable memory checking is to link your program with libmcheck:
如果你希望檢查整個堆而不是一個內存塊,你可以調用mcheck_check_all()來遍歷所有的活動塊。你也可以指導內存管理慣例,通過使用mcheck_check_all(),替代只檢查當前內存塊,同時使用mcheck_pedantic()而不是mcheck()來作初始化。注意,盡管這種方法是非常耗時的,第三種啟用內存檢查的方法是將你程序鏈接到libmcheck():
gcc myprog.c -o myprog -lmcheck
The mcheck() function is called automatically before the first memory allocation takes place—useful in those cases when some dynamic blocks are allocated before entering main().
在第一內存分配發生前,mcheck()函數被自動調用--當在進入main()之前,動態內存塊被分配的情況下,非常有用。 Memory Debugging: Tracing Blocks 內存調試:跟蹤內存塊
Tracing the history of memory blocks helps in finding problems related to memory leaks and usage or release of already freed blocks. For this purpose, the GNU C library offers a tracing facility that is enabled by calling the mtrace() function. Once this call is made, every heap operation is logged to a file whose name must be specified in the environment variable MALLOC_TRACE. Analysis of the log file then can be performed off-line using a Perl script that is provided with the library and called, not surprisingly, mtrace. Logging can be stopped by calling muntrace(), but keep in mind that applying tracing to portions of your program may invalidate the result of post-processing. For example, false leaks may be detected if you allocate one block while tracing and then free it after muntrace().
跟蹤內存塊的歷史有助于尋找到內存泄露和使用或釋放一個已經釋放的內存塊相關的問題。為了這個目的,GNU C庫提供了一個跟蹤工具,它通過調用mtrace()生效。一旦調用了這個函數,每次堆操作將被記錄到一個文件,此文件的名字必須在環境變量MALLOC_TRACE中指定。然后就可以在線下通過使用由庫一起提供的一個Perl腳本和調用mtrace來完成日志文件分析。日志能夠通過調用muntrace()函數來終止,但是時刻注意,在你的部分程序中應用內存跟蹤可能導致你后面的處理結果失效。例如,如果你在跟蹤期間分配一個內存塊,然后在muntrace()之后釋放它,就可能導致探測到一個虛假的內存泄露。
Listing 3. Tracing with mtrace()
使用mtrace()跟蹤
Here is a sample tracing session using the program in Listing 3:
這里有一個使用跟蹤的簡單程序:
Memory tracing has nothing to do with protection from errors; calling mtrace() won't prevent the program from crashing. Even worse, if the program segfaults, the trace file is likely to be truncated and tracing may be inconsistent. To protect against this risk, it is always a good idea to install a SIGSEGV handler that calls muntrace(), because it closes the trace file before aborting (Listing 4). More information on memory tracing can be found on the libc info page.
內存跟蹤與錯誤保護沒有什么聯系。調用mtrace()不會避免程序崩潰。甚至更糟的是,如果程序段錯誤,跟蹤文件可能被截掉,并且跟蹤也不是連續的。為了避免這個風險,安裝一個SIGSEGV信號處理器是一個不錯的主意,這個處理器會調用muntrace() ,因為它在終止程序之前關閉跟蹤文件(第4節)。更多關于內存跟蹤的信息可以在libc info page 上面找到。
Listing 4. Remember to call muntrace() in the SIGSEGV handler.
記著在SIGSEGV處理器中調用muntrace() Debugging Internals 內部調試
Sometimes the standard debugging facilities provided by the GNU C library may not be suited to the particular needs of your program. In this case, you can resort either to an external memory debugging tool (see Resources) or carve your own inside the library. Doing this is simply a matter of writing three functions and hooking them to these predefined variables:
一些GNU C庫提供的標準調試工具可能并不適合你程序的特殊需求。在這種情況下,你可以借助一個外部的內存調試工具(見 Resource)或者在你的庫內部作修改。做這件事中只是簡單的寫三個函數以及將它們與預先定義的變量相關聯:
Hooks also are available for other memory-related calls, including realloc(), calloc() and so on. Be sure to save the previous values of the hooks and restore them before calling malloc() or free() inside your routines. If you fail to do so, infinite recursion prevents your code from working. Have a look at the example given in the libc info page for memory debugging to see all the nifty details.
在其它的內存相關的調用中,Hooks()也有效,包括realloc(),calloc()等等。確保在調用malloc()或free()之前,保存先前的勾子的值,把它們存儲起來。如果你不這么做,你的程序將陷入無盡的遞歸。看看libc info page給的一個內存調試的例子來看看相關細節。
As a final note, consider that these hooks also are used by the mcheck and mtrace systems. It's a good idea to be careful when using all of them combined.
最后一點,勾子也被mcheck和mtrace系統使用。在使用所有它們的組合的時候,小心是沒錯的。
Conclusions
結語
The GNU C library offers several extensions that turn out to be quite useful when dealing with memory. If you want to fine-tune your application's memory usage or build a memory debugging solution tailored to your needs, you probably will find these tools helpful or, at least, a good starting point to develop your own mechanisms.
GNU C庫提供幾種擴展,在處理內存時,它們被證明是非常有用的。如果你希望很好的調整你程序的內存使用,或是構建一個內存調試解決方法使之適合你的需求,你可能將會發現這些工具很有用,或者,至少是開發你自己的機制時一個不錯的出發點。
總結
以上是生活随笔為你收集整理的Advanced Memory Allocation 内存分配进阶的全部內容,希望文章能夠幫你解決所遇到的問題。
- 上一篇: Linux下glibc内存管理
- 下一篇: 深度学习相关资料总结