GoldenGate – using FILTER, COMPUTE and SQLEXEC commands

Some time back I had posted a note on column mapping and data transformation using GoldenGate.

Here are some more examples of column mapping and manipulating data using keywords like SQLPREDICATE, COMPUTE, FILTER and I will also introduce another powerful GoldenGate command called SQLEXEC – which we will discuss in detail at a later date.

SQLPREDICATE

Enables us to provide a WHERE clause to select rows for an initial load. This will be included in the Extract parameter file as part of the TABLE clause as shown below.

The GoldenGate reference guide has this to say ….

“SQLPREDICATE is a better selection method for initial loads than the WHERE or FILTER options.It is much faster because it affects the SQL statement directly and does not require GoldenGate to fetch all records before filtering them, like those other options do.”

TABLE ggs_owner.emp_details, SQLPREDICATE “where ename=’Gavin’”;

We can also perform the filter on the Replicat side by only selecting a subset of the data which has been extracted by using the WHERE clause as part or the Replicat parameter file as shown below.

MAP ggs_owner.emp_details, TARGET ggs_owner.emp_details, WHERE (ename=”Gavin”);

FILTER

The FILTER clause offers us more functionality than the WHERE clause because you can employ any of GoldenGate’s column conversion functions to filter data, whereas the WHERE clause accepts basic WHERE operators.

For example we can use standard arithmetic operators like ‘+’,'-’,'/’,'*’ or comparison operators like ‘>’,’<', '=' as well as GoldenGate functions like @COMPUTE, @DATE, @STRFIND, @STRNUM etc

For example we can use the STRFIND function as part of the Extract parameter file to only extract records from the table that match a particular string value as shown below.

TABLE ggs_owner.emp_details,FILTER (@STRFIND (ename, “Gavin”) > 0);

COMPUTE

In this example we will use the GoldenGate function @COMPUTE to derive the values for a column in a table based on values in some other column in the same table.

We will also see how a column mapping is performed on the target side where the target table EMP has an additional column COMM which is not present in the source table. We will derive the values for the COMM column by using a arithmetical calculation where the COMM is the SAL value plus 10%.

Remember we have to first create a definitions file using the defgen command as the source and target tables differ in structure.

In this case we will generate the definitions file on the target GoldenGate environment as the target table has the additional column COMM which is not present in the source EMP table.

edit params defgen

DEFSFILE /home/oracle/goldengate/dirsql/emp.sql
USERID ggs_owner, PASSWORD ggs_owner
TABLE ggs_owner.emp;

We then run this on the Target goldengate location

[oracle@linux02 goldengate]$ ./defgen paramfile /home/oracle/goldengate/dirprm/defgen.prm

The replicat parameter file will have the following contents – note the combination of the COLMAP and COMPUTE – one will tell Goldengate how to map the difference in the table structure and the other will execute a computation on the SAL column to derive data for the COMM column. Remember, the USEDEFAULTS term means that all the other columns other than COMM are identically matched on both source as well as target tables.

REPLICAT rep1
USERID ggs_owner, PASSWORD *********
SOURCEDEFS /home/oracle/goldengate/dirsql/emp.sql
MAP ggs_owner.emp_details, TARGET ggs_owner.emp_details,
COLMAP (usedefaults,
comm= @compute(sal +sal *.10));

After running the extract on the source, we will see that the EMP table has been populated on the target database and the column COMM has been derived as well from the SAL column.

SQL> select * from emp;

     EMPNO ENAME                    DEPTNO        SAL       COMM
---------- -------------------- ---------- ---------- ----------
      1001 Gavin                        10       1000       1100
      1002 Mark                         20       2000       2200
      1003 John                         30       3000       3300

SQLEXEC

SQLEXEC can be used as part of the Extract or Replicat process to make database calls which enables Goldengate to use the native SQL of the database to execute SQL queries, database commands as well as stored procedures and functions.

For example, as part of a large batch data load process, we would like to drop the indexes first and then rebuild them after the data load is complete. We see how when included as part of this replicat parameter file we are dropping and rebuilding an index using native SQL commands.

REPLICAT rep1
USERID ggs_owner, PASSWORD ggs_owner
ASSUMETARGETDEFS
sqlexec “drop index loc_ind”;
MAP ggs_owner.emp_details, TARGET ggs_owner.emp_details, WHERE (location=”Sydney”);
sqlexec “create index loc_ind on emp_details(location)”;