GoldenGate real-time replication from Active Standby Database to SQL Server 2012 target
GoldenGate Initial Load Methods Oracle source to SQL Server 2012 target
GoldenGate change data capture and replication of BLOB and CLOB data
We will look at an example of GoldenGate replication of a table having a BLOB column and how an INSERT and UPDATE statement on the table with BLOB data is handled by GoldenGate.
We create an APEX 4.2 application to illustrate this example where we create a form and report based on the DOCUMENTS table and upload and download documents. We will observe how changes to the BLOB column data are replicated in real-time to the target database via GoldenGate change data capture.
Thanks to ACE Director Eddie Awad’s article which made me understand how APEX handles file upload and downloads.
Read the article by Eddie.
On the source database we create the DOCUMENTS table and a sequence and trigger to populate the primary key column ID.
CREATE TABLE documents ( ID NUMBER PRIMARY KEY ,DOC_CONTENT BLOB ,MIME_TYPE VARCHAR2 (255) ,FILENAME VARCHAR2 (255) ,LAST_UPDATED DATE ,CHARACTER_SET VARCHAR2 (128) ); CREATE SEQUENCE documents_seq; CREATE OR REPLACE TRIGGER documents_trg_bi BEFORE INSERT ON documents FOR EACH ROW BEGIN :new.id := documents_seq.NEXTVAL; END; /
Create the Extract and Replicat processes
GGSCI (vindi-a) 3> add extract ext9 tranlog begin now EXTRACT added. GGSCI (vindi-a) 4> add rmttrail /u01/app/oracle/product/st_goldengate/dirdat/xx extract ext9 RMTTRAIL added. GGSCI (vindi-a) 5> edit params ext9 extract ext9 CACHEMGR CACHESIZE 8G userid gg_owner@testdb password gg_owner DDL include ALL ddloptions addtrandata, report rmthost poc-strelis-vindi, mgrport 7810 rmttrail /u01/app/oracle/product/st_goldengate/dirdat/xx dynamicresolution SEQUENCE GG_OWNER.*; TABLE GG_OWNER.DOCUMENTS; GGSCI (vindi-a) 1> start extract ext9 Sending START request to MANAGER ... EXTRACT EXT9 starting GGSCI (vindi-a) 2> info extract ext9 EXTRACT EXT9 Last Started 2014-05-22 08:43 Status RUNNING Checkpoint Lag 00:00:00 (updated 00:04:54 ago) Process ID 17984 Log Read Checkpoint Oracle Redo Logs 2014-05-22 08:38:44 Seqno 122, RBA 9039376 SCN 0.0 (0) On Target GoldenGate GGSCI (vindi-a) 3> add replicat rep9 exttrail /u01/app/oracle/product/st_goldengate/dirdat/xx REPLICAT added. GGSCI (vindi-a) 4> edit params rep9 replicat rep9 assumetargetdefs ddlerror default ignore userid gg_owner@strelis password gg_owner MAP GG_OWNER.DOCUMENTS ,TARGET GG_OWNER.DOCUMENTS; GGSCI (vindi-a) 5> start replicat rep9 Sending START request to MANAGER ... REPLICAT REP9 starting GGSCI (vindi-a) 6> info replicat rep9 REPLICAT REP9 Last Started 2014-05-22 08:43 Status RUNNING Checkpoint Lag 00:00:00 (updated 00:00:04 ago) Process ID 17919 Log Read Checkpoint File /u01/app/oracle/product/st_goldengate/dirdat/xx000000 First Record RBA 0
Since we have configured DDL replication as well we see that the DOCUMENTS table has been created on the target database as well.
oracle@vind-a:/export/home/oracle $ sqlplus gg_owner/gg_owner@targetdb SQL*Plus: Release 11.2.0.3.0 Production on Thu May 22 08:35:55 2014 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options SQL> desc documents Name Null? Type ----------------------------------------- -------- ---------------------------- ID NOT NULL NUMBER DOC_CONTENT BLOB MIME_TYPE VARCHAR2(255) FILENAME VARCHAR2(255) LAST_UPDATED DATE CHARACTER_SET VARCHAR2(128)
We now launch APEX to create our demo application.
In Application Builder click on Create
Select Database
Click Add Page
Click Next
Accept default value
Accept default values
Click Create Application
Click Page 1 link
From Regions menu click Create
Select Form and click Next
Select Form on a Table or View
Select DOCUMENTS table from LOV and click Next
Enter the Page name and click Next
Select the primary key column of the DOCUMENTS table and click Next
Primary key of the table is populated via the sequence called by the trigger
Select Existing trigger and click Next
Select columns to display on the form and click Next
Change the label of the Create button to Upload and hide other buttons – click Next
Select the current page as the page to branch to and click Next
Click Create
Click Edit Page
Select the P1_DOC_CONTENT item and click Edit from the menu
In the Settings section of the page add the table column names against the columns as shown above
Click on Apply Changes
Click on Run
Enter workspace or application login credentials
Click the Browse button and select the file to upload
Click Upload
In GoldenGate we check the extract and replication stats and we can see the capture and apply of the change we just made
GGSCI (vind-a) 3> stats extract ext9 latest Sending STATS request to EXTRACT EXT9 ... Start of Statistics at 2014-05-22 09:01:44. DDL replication statistics (for all trails): *** Total statistics since extract started *** Operations 0.00 Mapped operations 0.00 Unmapped operations 0.00 Other operations 0.00 Excluded operations 0.00 Output to /u01/app/oracle/product/st_goldengate/dirdat/xx: Extracting from GG_OWNER.DOCUMENTS_SEQ to GG_OWNER.DOCUMENTS_SEQ: *** Latest statistics since 2014-05-22 08:59:16 *** Total updates 1.00 Total discards 0.00 Total operations 1.00 Extracting from GG_OWNER.DOCUMENTS to GG_OWNER.DOCUMENTS: *** Latest statistics since 2014-05-22 08:59:16 *** Total inserts 1.00 Total updates 1.00 Total deletes 0.00 Total discards 0.00 Total operations 2.00 End of Statistics.
Connect to the target database and see if the record has been inserted.
Note the colums like MIME_TYPE, LAST_UPDATED, FILENAME etc are automatically populated.
SQL> col filename format a60 SQL> col mime_type format a30 SQL> set linesize 120 SQL> select id,filename,mime_type from documents; ID FILENAME MIME_TYPE ---------- ------------------------------------------------------------ ------------------------------ 1 Consultant Profile - Gavin Soorma.doc application/msword
Note the size of the document
SQL> select id,filename,dbms_lob.getlength(doc_content) from documents; ID FILENAME DBMS_LOB.GETLENGTH(DOC_CONTENT) ---------- ------------------------------------------------------------ ------------------------------- 1 Consultant Profile - Gavin Soorma.doc 532992
We will now add a new page to the application. Click Create Page
Select Form and click Next
Select Form on a Table with Report and click Next
Change the Region Title to Edit Documents and click Next
Select the table and click Next
Give a name for the tab for the new page we are creating and click Next
Select the columns to include in the report and click Next
Accept default and click Next
Change the Region Title and click Next
Select Primary key column and click Next
Select the columns to include in the form and click Next
Click Create
Click Run Page
Click the Edit icon
Click the Edit Page link at the bottom of the page
In the Settings section of the page add the table column names as shown
Click Apply Changes and then Run
We will now download the document from the table, edit the document and upload it back into the database again
Click on the Download link and save the document
Open the document and we will make some changes
We will delete the “Technical Skills” table from the document, save it and then upload it back again
Click on Browse and upload the document which we just downloaded and edited
Click Apply Changes
Oracle GoldenGate has applied this change and we can see that the size of the document has reduced in the target database from 532992 bytes to 528896 bytes as we had deleted some lines from the document.
Connect to the target database and issue the query
Previous: SQL> select id,filename,dbms_lob.getlength(doc_content) from documents; ID FILENAME DBMS_LOB.GETLENGTH(DOC_CONTENT) ---------- ------------------------------------------------------------ ------------------------------- 1 Consultant Profile - Gavin Soorma.doc 532992 Current: SQL> select id,filename,dbms_lob.getlength(doc_content) from documents; ID FILENAME DBMS_LOB.GETLENGTH(DOC_CONTENT) ---------- ------------------------------------------------------------ ------------------------------- 1 Consultant Profile - Gavin Soorma.doc 528896
We can see that the Replicat process which had earlier applied the INSERT statement when the document was uploaded to the database the first time has now applied some UPDATE statements as well
GGSCI (vind-a) 1> stats replicat rep9 latest Sending STATS request to REPLICAT REP9 ... Start of Statistics at 2014-05-22 10:08:47. Replicating from GG_OWNER.DOCUMENTS to GG_OWNER.DOCUMENTS: *** Latest statistics since 2014-05-22 08:59:20 *** Total inserts 1.00 Total updates 3.00 Total deletes 0.00 Total discards 0.00 Total operations 4.00
GoldenGate and Virtual Memory – CACHEMGR CACHESIZE and CACHEDIRECTORY
GoldenGate Bounded Recovery
GoldenGate Director Security
Platform Migration and Database Upgrade from Oracle 9i to Oracle 11g using GoldenGate
Oracle Goldengate 12c on DBFS for RAC and Exadata
GoldenGate 12c (12.2) New Features
Goldengate 12.2 New Feature Self-describing Trail Files
GoldenGate 12.2 supports INVISIBLE columns
Tuning Integrated Replicat performance using EAGER_SIZE parameter
Goldengate 12.2 New Feature – Check and validate parameter files using chkprm
In GoldenGate 12.2 we can now validate parameter files before deployment.
There is a new utility called chkprm which can be used for this purpose.’
To run the chkprm utility we provide the name of the parameter file and can optionally indicate what process this parameter file belongs to using the COMPONENT keyword.
Let us look at an example.
Register & Purchase Purchase Only Login & Purchase
Oracle GoldenGate 12.2 New Feature – Integration with Oracle Datapump
How to configure high availability for Oracle GoldenGate on Exadata
GoldenGate INSERTALLRECORDS and OGG-01154 SQL error 1400
Installing the Oracle GoldenGate monitoring plug-in (13.2.1.0.0) for Cloud Control 13c Release 2
Installing and Configuring Oracle GoldenGate Monitor 12c (12.1.3.0)
Oracle GoldenGate on Oracle Cloud Marketplace (Part 1)
Oracle Cloud Marketplace (https://cloudmarketplace.oracle.com) is an online store selling a wide variety of business applications and services which can complement an existing Oracle Cloud implementation. All these applications and services available on the Cloud marketplace are offered by Oracle approved and registered partners and developers.
Oracle GoldenGate is one of the many applications available on Oracle Cloud Marketplace and described very simply, it is the cloud-based version of Oracle GoldenGate that also provides us with all the core functionality of GoldenGate as a product including the scalability, reliability and manageability.
After we provision GoldenGate on Cloud Marketplace, we have access to a compute node which includes an image of the latest 19c Oracle GoldenGate software (19.1.0.0.4) as well as Oracle Database Client software (11g, 12g, 18c, and 19c) pre-installed.
GoldenGate on Cloud Marketplace is available with the choice of deploying using Classic Architecture as well as Microservices Architecture.
This provides us a replication solution from not only on-premise to the cloud as well as cloud-to-cloud platform.
We need an Oracle Cloud Account and access to compute node resources within Oracle Cloud Tenant. Supported shapes are VM.Standard2.4, VM.Standard2.8, VM.Standard2.16 and VM.Standard2.24.
In addition, we also need to provide an SSH Public Key which will enable us to connect to the Oracle Cloud Infrastructure hosted on the GoldenGate compute node as the opc user.
In this example we will be configuring real-time replication from an Oracle 12c Release 1 database on OCI-Classic (Gen 1) to an Oracle 18c database hosted on OCI ATP (Gen 2) using Oracle GoldenGate running on Oracle Cloud Marketplace.
In Part 1, we will look at the various screen shots from the process of deploying Oracle GoldenGate Microservices on Oracle Cloud Marketplace.
Oracle GoldenGate on Oracle Cloud Marketplace (Part 2)
Connect to the GoldenGate Cloud Marketplace Node
Identify the Public IP Address of the GoldenGate Cloud Marketplace compute node and connect via PuTTY using the saved PPK private key. (note we had to provide the SSH Public Key details when we deployed Oracle GoldenGate Cloud Marketplace)



Note the directory structure of the GoldenGate Cloud Marketplace Node
The trail files are stored in the directory /u02/trails and we have the two deployments (‘Source’ and ‘Target’) related directories under /u02/deployments


The GoldenGate software is available under /u01/app/ogg. Note the different versions of the software available.

Change the oggadmin password
Open the ogg-credentials.json file located in the /home/opc directory. Use the credentials included in that file to connect via Service Manager URL as oggadmin user.
-bash-4.2$ pwd
/home/opc
-bash-4.2$ ls -l
total 0
lrwxrwxrwx. 1 opc opc 37 Mar 26 05:22 ogg-credentials.json -> /u02/deployments/ogg-credentials.json
-bash-4.2$ cat ogg-credentials.json
{“username”: “oggadmin”, “credential”: “k%KaBo.2XDmIFRCa”}






Copy wallet zip file which has been downloaded from ATP database
This is required so we can connect to the ATP target database from the GoldenGate Cloud Marketplace compute node.
-bash-4.2$ mkdir wallet
-bash-4.2$ ls
ogg-credentials.json wallet Wallet_GSATP.zip
-bash-4.2$ cp Wallet_GSATP.zip ./wallet/
-bash-4.2$ cd wallet
-bash-4.2$ unzip Wallet_GSATP.zip
Archive: Wallet_GSATP.zip
inflating: cwallet.sso
inflating: tnsnames.ora
inflating: readme.md
inflating: truststore.jks
inflating: ojdbc.properties
inflating: sqlnet.ora
inflating: ewallet.p12
inflating: keystore.jks
Configure Network Connectivity to source OCI Classic database
Edit the network configuration files for deployment “Source”.
Note: replace the host name with the public IP address .
-bash-4.2$ pwd
/u02/deployments/Source/etc
-bash-4.2$ vi tnsnames.ora
ORCL =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 150.136.58.202)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ORCL.svcsubnetad1.svcvcn.oraclevcn.com)
)
)
PDB1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 150.136.58.202)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = pdb1.svcsubnetad1.svcvcn.oraclevcn.com)
)
)
Configure source OCI Classic Database for GoldenGate Replication
SQL> SELECT supplemental_log_data_min, force_logging FROM v$database;
SUPPLEME FORCE_LOGGING
——– —————————————
YES YES
SQL> show parameter goldengate
NAME TYPE VALUE
———————————— ———– ——————————
enable_goldengate_replication boolean TRUE
SQL> alter system set streams_pool_size=1024m scope=both;
System altered.
SQL> select username from dba_users where username like ‘%GG%’;
USERNAME
——————————————————————————–
C##GGADMIN
SQL> conn C##GGADMIN/Dreamliner787##
Connected.
Update Credential Store with source database credentials
Enter the credentials for the common user c##oggadmin which connects to the source container database ORCL.





Add SCHEMATRANDATA



Configure connectivity to ATP target database from GoldenGate Cloud Marketplace node
-bash-4.2$ pwd
/u02/deployments/Target/etc
-bash-4.2$ vi sqlnet.ora
WALLET_LOCATION = (SOURCE = (METHOD = file) (METHOD_DATA = (DIRECTORY=”/home/opc/wallet”)))
SSL_SERVER_DN_MATCH=yes
-bash-4.2$ cat tnsnames.ora
gsatp_high = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=adb.us-ashburn-1.oraclecloud.com))(connect_data=(service_name=zjiver6hk5luiwo_gsatp_high.atp.oraclecloud.com))(security=(ssl_server_cert_dn=”CN=adwc.uscom-east-1.oraclecloud.com,OU=Oracle BMCS US,O=Oracle Corporation,L=Redwood City,ST=California,C=US”)))
gsatp_low = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=adb.us-ashburn-1.oraclecloud.com))(connect_data=(service_name=zjiver6hk5luiwo_gsatp_low.atp.oraclecloud.com))(security=(ssl_server_cert_dn=”CN=adwc.uscom-east-1.oraclecloud.com,OU=Oracle BMCS US,O=Oracle Corporation,L=Redwood City,ST=California,C=US”)))
gsatp_medium = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=adb.us-ashburn-1.oraclecloud.com))(connect_data=(service_name=zjiver6hk5luiwo_gsatp_medium.atp.oraclecloud.com))(security=(ssl_server_cert_dn=”CN=adwc.uscom-east-1.oraclecloud.com,OU=Oracle BMCS US,O=Oracle Corporation,L=Redwood City,ST=California,C=US”)))
gsatp_tp = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=adb.us-ashburn-1.oraclecloud.com))(connect_data=(service_name=zjiver6hk5luiwo_gsatp_tp.atp.oraclecloud.com))(security=(ssl_server_cert_dn=”CN=adwc.uscom-east-1.oraclecloud.com,OU=Oracle BMCS US,O=Oracle Corporation,L=Redwood City,ST=California,C=US”)))
gsatp_tpurgent = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=adb.us-ashburn-1.oraclecloud.com))(connect_data=(service_name=zjiver6hk5luiwo_gsatp_tpurgent.atp.oraclecloud.com))(security=(ssl_server_cert_dn=”CN=adwc.uscom-east-1.oraclecloud.com,OU=Oracle BMCS US,O=Oracle Corporation,L=Redwood City,ST=California,C=US”)))
-bash-4.2$ pwd
/u01/app/client/oracle19/network/admin
-bash-4.2$ cp /u02/deployments/Target/etc/sqlnet.ora .
-bash-4.2$ cp /u02/deployments/Target/etc/tnsnames.ora .
Unlock the ggadmin user in the ATP database and grant privileges on DATA tablespace
-bash-4.2$ export TNS_ADMIN=/u01/app/client/oracle19/network/admin
-bash-4.2$ cd ../..
-bash-4.2$ cd bin
-bash-4.2$ export ORACLE_HOME=/u01/app/client/oracle19
-bash-4.2$ export PATH=$ORACLE_HOME/bin:$PATH
-bash-4.2$ sqlplus admin/Dreamliner787##@gsatp_high
SQL*Plus: Release 18.0.0.0.0 – Production on Wed Apr 1 01:45:29 2020
Version 18.3.0.0.0
Copyright (c) 1982, 2018, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 – Production
Version 19.5.0.0.0
SQL> alter user ggadmin identified by Dreamliner787## ;
User altered.
SQL> alter user ggadmin account unlock;
User altered.
SQL> alter user ggadmin quota unlimited on data;
User altered
Create the directory to store the trail files for each deployment
bash-4.2$ cd /u02/
-bash-4.2$ ls
cacheManager deployments trails
-bash-4.2$ cd trails/
–bash-4.2$ mkdir Source
-bash-4.2$ mkdir Target