After a recent Oracle GoldenGate installation at a client site running on Solaris 11, we observed memory related errors in the GoldenGate error log like the ones mentioned below as well as extract processes were abending on start up.
ERROR OGG-01841 CACHESIZE TOO SMALL:
ERROR OGG-01843 default maximum buffer size (2097152) > absolute maximum buffer size (0)
The Oracle database alert log also seemed to be reporting quite few “ORA-04030: out of process memory” type errors as well.
The Solaris server had 64 GB of RAM but it seemed that GoldenGate was requiring 128 GB when the extract processes were started.
Let us see why and take a look also at how GoldenGate manages memory operations.
The Oracle redo log files contain both committed as well as uncommitted changes but GoldenGate only replicates committed transactions. So it needs some kind of cache where it can store the operation of each transaction until it receives a commit or rollback for that transaction. This is particularly significant for both large as well as long-running transactions.
This cache is a virtual memory pool or global cache for all the extract and replicat processes and sub-pools are allocated for each Extract log reader thread or Replicat trail reader thread as well as dedicated sub-pools for holding large data like BLOBs.
Documentation states: “While the actual amount of physical memory that is used by any Oracle GoldenGate process is controlled by the operating system, the cache manager keeps an Oracle GoldenGate process working within the soft limit of its global cache size, only allocating virtual memory on demand.”
The CACHEMGR parameter controls the amount of virtual memory and temporary disk space that is available for caching uncommitted transaction data.
The CACHEMGR CACHESIZE parameter controls the virtual memory allocations and in GoldenGate versions 11.2 onwards for a 64-bit system the CACHESIZE by default is 64 GB.
While the CACHESIZE parameter controls the Virtual Memory, if that is exceeded then GoldenGate will swap data to disk temporarily and that is by default being allocated in the dirtmp sub-directory of the Oracle GoldenGate installation directory.
The dirtmp location will contain the .cm files. The cache manager assumes that all of the free space on the file system is available and will use it to create the .cm files until it becomes full. To regulate this we can use the CACHEMGR CACHEDIRECTORY parameter and provide both a size as well assign a directory location where these .cm files will be created.
So the usage of these parameters are:
CACHEMGR CACHESIZE {size}
CACHEMGR CACHEDIRECTORY {path} {size}
The CACHESIZE as mentioned earlier on 64-bit systems defaults to 64 GB and we see that 128 GB is being used because the documentation states:
“The CACHESIZE value will always be a power of two, rounded down from the value of PROCESS VM AVAIL FROM OS”
So in our case we had set the extract and replicat proceses to be started automatically by the manager on restart. These processes start simultaneously so when the first extract process started it momentarily grabbed 128 GB of memory and there was no memory left for the other extract processes to start.
So we used the CACHESIZE parameter to regulate the upper limit on which machine virtual memory can be used by GoldenGate by adding this parameter to each of the extract parameter files:
CACHEMGR CACHESIZE 8G