null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null lkm_rowbyrow.log]]> null null null null null null null null null null null null null null null null null null null null null null null null null null null ' logLevel = <%=odiRef.getOption("LOG_LEVEL")%> maxErrors= <%=odiRef.getOption("MAX_ERRORS")%> # set the logger logger = Log(logfilename=logFileName, loglevel=logLevel) try: # Source Connection ConSrc = odiRef.getJDBCConnection( "SRC" ) # Target (Destination) Connection ConTrg = odiRef.getJDBCConnection( "DEST" ) logger.record_time('get metadata information from target', LOG_ACTION) # Statement target + metadata stmtTrg = ConTrg.createStatement() readTrg = stmtTrg.executeQuery("select * from <%=odiRef.getTable("L", "INT_NAME", "W")%> where 1=2") metaDataTrg = readTrg.getMetaData() colLst = '' typeColTrg='' bindVar ='' colLst= """<%=odiRef.getColList("", "[COL_NAME]", ",\n\t","", "((INS and !TRG) and REW)")%> """ nbCol=colLst.count(',') bindVar=','.join('?'*(nbCol+1)) beginRqteInsert = 'insert into <%=odiRef.getTable("L","TARG_NAME","A")%> ('+colLst+') values (' rqteInsert = beginRqteInsert+bindVar+')' psmt = ConTrg.prepareStatement(rqteInsert) # Statement source + metadata readSrc = ConSrc.createStatement() sqlSrc="""select <%=odiRef.getColList("", "[COL_NAME]", ",\n\t", "", "((INS and !TRG) and REW)")%> <%=odiRef.getColList(",", "[EXPRESSION]", ",\n\t", "", "((INS and TRG) and REW)")%> from <%=odiRef.getTable("L","INT_NAME","A")%> where IND_UPDATE = 'I'""" readSrc.setFetchSize(100) logger.record_time('Select action on source', LOG_ACTION) logger.record_time(sqlSrc, LOG_ACTION) rqteSrc = readSrc.executeQuery(sqlSrc) metaDataSrc = rqteSrc.getMetaData() # Set statement nb = None record = '' nbLine = 0 nbErrors = 0 psmt.clearBatch() while rqteSrc.next(): resLine='' record='' for nb in range (nbCol+1): colType=metaDataSrc.getColumnType(nb+1) if colType == sql.Types.VARCHAR: #Code 12 resline=rqteSrc.getString(nb+1) psmt.setString(nb+1,resline) elif colType == sql.Types.CHAR: #Code 1 resline=rqteSrc.getString(nb+1) psmt.setString(nb+1,resline) elif colType == sql.Types.INTEGER: #Code 4 resline=rqteSrc.getInt(nb+1) psmt.setInt(nb+1,resline) elif colType == sql.Types.NUMERIC: #Code 2 resline=rqteSrc.getBigDecimal(nb+1) psmt.setBigDecimal(nb+1,resline) elif colType == sql.Types.DECIMAL: #Code 3 resline=rqteSrc.getBigDecimal(nb+1) psmt.setBigDecimal(nb+1,resline) elif colType == sql.Types.DATE: #Code 91 resline=rqteSrc.getDate(nb+1) psmt.setDate(nb+1,resline) elif colType == sql.Types.TIME: #Code 92 resline=rqteSrc.getTime(nb+1) psmt.setTime(nb+1,resline) elif colType == sql.Types.TIMESTAMP: #Code 93 resline=rqteSrc.getTimestamp(nb+1) psmt.setTimestamp(nb+1,resline) elif colType == sql.Types.BIGINT: #Code -5 resline=rqteSrc.getLong(nb+1) psmt.setLong(nb+1,resline) elif colType == sql.Types.BINARY: #Code -2 resline=rqteSrc.getBytes(nb+1) psmt.setBytes(nb+1,resline) elif colType == sql.Types.BIT: #Code -7 resline=rqteSrc.getBoolean(nb+1) psmt.setBoolean(nb+1,resline) elif colType == sql.Types.DOUBLE: #Code 8 resline=rqteSrc.getDouble(nb+1) psmt.setDouble(nb+1,resline) elif colType == sql.Types.LONGVARBINARY: #Code -4 resline=rqteSrc.getBytes(nb+1) psmt.setBytes(nb+1,resline) elif colType == sql.Types.LONGVARCHAR: #Code -1 resline= rqteSrc.getString(nb+1) psmt.setString(nb+1,resline) elif colType == sql.Types.OTHER: #Code 1111 resline=rqteSrc.getObject(nb+1) psmt.setObject(nb+1,resline) elif colType == sql.Types.REAL: #Code 7 resline=rqteSrc.getFloat(nb+1) psmt.setFloat(nb+1,resline) elif colType == sql.Types.SMALLINT: #Code 5 resline=rqteSrc.getShort(nb+1) psmt.setShort(nb+1,resline) elif colType == sql.Types.TINYINT: #Code -6 resline=rqteSrc.getByte(nb+1) psmt.setByte(nb+1,resline) elif colType == sql.Types.VARBINARY: #Code -3 resline=rqteSrc.getBytes(nb+1) psmt.setBytes(nb+1,resline) record=record+str(resline)+', ' resline = '' try: logger.record_raw(str(nbLine) +' : ' + beginRqteInsert+' '+record[:-2]+' )' , LOG_ALL) psmt.executeUpdate() except: logger.record_raw(str(nbLine) +' : ' + beginRqteInsert+' '+record[:-2]+' )' , LOG_ERROR) nbErrors+=1 if nbErrors >= maxErrors: raise "Too much errors in the Load step", "The error limit has been reached. A maximum of "+str(maxErrors)+" errors was allowed." nbLine=nbLine+1 finally: print 'OK' ]]> null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null null /* Consistant journalization detected. */ /* Lock should be done by a specific step in a package */ <% } else { %> delete from <%=odiRef.getTable("L","JRN_NAME","A")%> where JRN_CONSUMED = '1' <%=odiRef.getJrnFilter()%> <% } %>]]> null null null null null null null null null null null null null null null null null null null ]]> null null null null null null null null null null null null null null null null null null null null null /* Consistant journalization detected. */ /* Lock should be done by a specific step in a package. No operation here */ <% } else { %> update <%=odiRef.getTable( "L" , "JRN_NAME" , "A" )%> set JRN_CONSUMED = '1' where (1=1) <%=odiRef.getJrnFilter()%> <% } %>]]> null null null null null null null null null null null null null null null null null null ]]> null null null null null null null null null null null null null null null null null null null ]]> null null null null null null null null null null null null null null null null null null null ( <%=odiRef.getTargetColList("", "[COL_NAME]\t[DEST_CRE_DT] " + odiRef.getInfo("DEST_DDL_NULL"), ",\n\t", "")%> )]]> null null null null null null null null null null null null null null null null null null null ]]> null null null null null null null null null null null null null null null null null null null ( <%=odiRef.getColList("", "[COL_NAME]\t[DEST_WRI_DT] "+ odiRef.getInfo("DEST_DDL_NULL"),",\n\t", "", "")%>, IND_UPDATE <%=odiRef.getDataType("SRC_VARCHAR","1")%> )]]> null null null null null null null null null null null null null null null null null null null ]]> null null null null null null null null null null null null null null null null null null null null null set IND_UPDATE = 'N' where exists ( select 'X' from <%=odiRef.getTable("L", "TARG_NAME", "A")%> <%=odiRef.getInfo("DEST_TAB_ALIAS_WORD")%> T where <%=odiRef.getColList("", odiRef.getTable("L", "INT_NAME", "A")+".[COL_NAME]\t= T.[COL_NAME]", "\n\t\tand\t", "", "UK")%> <%=odiRef.getColList("and\t", "(("+odiRef.getTable("L", "INT_NAME", "A")+".[COL_NAME] = T.[COL_NAME]) or ("+odiRef.getTable("L", "INT_NAME", "A")+".[COL_NAME] IS NULL and T.[COL_NAME] IS NULL))", " \n\t\tand\t", "", "((UPD and !TRG) and !UK)")%> )]]> null null null null null null null null null null null null null null null ' logLevel = <%=odiRef.getOption("LOG_LEVEL")%> maxErrors= <%=odiRef.getOption("MAX_ERRORS")%> # set the logger logger = Log(logfilename=logFileName, loglevel=logLevel) def getDataSring(metaDataSrc, rqteSrc, nb, updStmt): colType=metaDataSrc.getColumnType(nb+1) if colType == sql.Types.VARCHAR: #Code 12 resline=rqteSrc.getString(nb+1) updStmt.setString(nb+1,resline) elif colType == sql.Types.CHAR: #Code 1 resline=rqteSrc.getString(nb+1) updStmt.setString(nb+1,resline) elif colType == sql.Types.INTEGER: #Code 4 resline=rqteSrc.getInt(nb+1) updStmt.setInt(nb+1,resline) elif colType == sql.Types.NUMERIC: #Code 2 resline=rqteSrc.getBigDecimal(nb+1) updStmt.setBigDecimal(nb+1,resline) elif colType == sql.Types.DECIMAL: #Code 3 resline=rqteSrc.getBigDecimal(nb+1) updStmt.setBigDecimal(nb+1,resline) elif colType == sql.Types.DATE: #Code 91 resline=rqteSrc.getDate(nb+1) updStmt.setDate(nb+1,resline) elif colType == sql.Types.TIME: #Code 92 resline=rqteSrc.getTime(nb+1) updStmt.setTime(nb+1,resline) elif colType == sql.Types.TIMESTAMP: #Code 93 resline=rqteSrc.getTimestamp(nb+1) updStmt.setTimestamp(nb+1,resline) elif colType == sql.Types.BIGINT: #Code -5 resline=rqteSrc.getLong(nb+1) updStmt.setLong(nb+1,resline) elif colType == sql.Types.BINARY: #Code -2 resline=rqteSrc.getBytes(nb+1) updStmt.setBytes(nb+1,resline) elif colType == sql.Types.BIT: #Code -7 resline=rqteSrc.getBoolean(nb+1) updStmt.setBoolean(nb+1,resline) elif colType == sql.Types.DOUBLE: #Code 8 resline=rqteSrc.getDouble(nb+1) updStmt.setDouble(nb+1,resline) elif colType == sql.Types.LONGVARBINARY: #Code -4 resline=rqteSrc.getBytes(nb+1) updStmt.setBytes(nb+1,resline) elif colType == sql.Types.LONGVARCHAR: #Code -1 resline= rqteSrc.getString(nb+1) updStmt.setString(nb+1,resline) elif colType == sql.Types.OTHER: #Code 1111 resline=rqteSrc.getObject(nb+1) updStmt.setObject(nb+1,resline) elif colType == sql.Types.REAL: #Code 7 resline=rqteSrc.getFloat(nb+1) updStmt.setFloat(nb+1,resline) elif colType == sql.Types.SMALLINT: #Code 5 resline=rqteSrc.getShort(nb+1) updStmt.setShort(nb+1,resline) elif colType == sql.Types.TINYINT: #Code -6 resline=rqteSrc.getByte(nb+1) updStmt.setByte(nb+1,resline) elif colType == sql.Types.VARBINARY: #Code -3 resline=rqteSrc.getBytes(nb+1) updStmt.setBytes(nb+1,resline) try: # Source Connection ConSrc = odiRef.getJDBCConnection( "SRC" ) # Target (Destination) Connection ConTrg = odiRef.getJDBCConnection( "DEST" ) logger.record_time('get metadata information from target', LOG_ACTION) # columns to update not in UK nbUpdColLst = <%=odiRef.getColList("", "1", "+", "", "(((UPD) and !TRG) and REW)")%> # Source statement + metadata readSrc = ConSrc.createStatement() sqlSrc="""select <%=odiRef.getColList("", "[COL_NAME]", ",\n\t", "", "((UPD and !TRG) and REW and !UK)")%>, <%=odiRef.getColList("", "[COL_NAME]", ",\n\t", "", "(UK)")%> from <%=odiRef.getTable("L", "INT_NAME", "A")%> where IND_UPDATE = 'U'""" readSrc.setFetchSize(100) logger.record_time('Select action on source', LOG_ACTION) logger.record_time(sqlSrc, LOG_ACTION) rqteSrc = readSrc.executeQuery(sqlSrc) metaDataSrc = rqteSrc.getMetaData() rqteUpdate = """update <%=odiRef.getTable("L", "TARG_NAME", "A")%> set <%=odiRef.getColList("", "[COL_NAME]\t= ?", ",\n\t", "", "(((UPD and !UK) and !TRG) and REW)")%> <%=odiRef.getColList(",", "[COL_NAME]\t= [EXPRESSION]", ",\n\t", "", "(((UPD and !UK) and TRG) and REW)")%> where <%=odiRef.getColList("", "[COL_NAME]\t= ?", "\nand\t", "", "(UK)")%>""" updStmt = ConTrg.prepareStatement(rqteUpdate) # Set statement nb = None record = '' nbLine = 0 nbErrors = 0 updStmt.clearBatch() while rqteSrc.next(): nbLine = nbLine + 1 resline='' for nb in range (nbUpdColLst): getDataSring(metaDataSrc, rqteSrc, nb, updStmt) try: logger.record_raw(str(nbLine) +' : ' + rqteUpdate, LOG_ALL) updStmt.executeUpdate() except: logger.record_raw(str(nbLine) +' : ' + rqteUpdate, LOG_ERROR) nbErrors+=1 if nbErrors >= maxErrors: raise "Too much errors in the Load step", "The error limit has been reached. A maximum of "+str(maxErrors)+" errors was allowed." finally: print 'OK' ]]> null null null null null null null null null null null null null null null null null null null null > f, l f.close() if self.printtoscreen: print l def record_raw(self, rec, level): if level <= self.loglevel: l = LogStr[level] + ': ' + rec f = open(self.logfilename, "a+") print >> f, l f.close() if self.printtoscreen: print rec ]]> null null null null null null null null null null null null null null null null null null ' logLevel = <%=odiRef.getOption("LOG_LEVEL")%> maxErrors= <%=odiRef.getOption("MAX_ERRORS")%> # set the logger logger = Log(logfilename=logFileName, loglevel=logLevel) try: # Source Connection ConSrc = odiRef.getJDBCConnection( "SRC" ) # Target (Destination) Connection ConTrg = odiRef.getJDBCConnection( "DEST" ) logger.record_time('get metadata information from target', LOG_ACTION) # Statement target + metadata stmtTrg = ConTrg.createStatement() readTrg = stmtTrg.executeQuery("select * from <%=odiRef.getTable("L", "INT_NAME", "W")%> where 1=2") metaDataTrg = readTrg.getMetaData() colLst = '' typeColTrg='' bindVar ='' for i in range(metaDataTrg.getColumnCount()): colLst = str(colLst) + str(metaDataTrg.getColumnName(i+1)) + ',' typeColTrg = str(typeColTrg)+ str(metaDataTrg.getColumnType(i+1))+',' colLst=colLst[:-1] colLst= """<%=odiRef.getColList("", "[COL_NAME]", ",\n\t", "", "(((INS or UPD) and !TRG) and REW)")%>, IND_UPDATE""" nbCol=colLst.count(',') bindVar=','.join('?'*(nbCol+1)) beginRqteInsert = 'insert into <%=odiRef.getTable("L","INT_NAME","A")%> ('+colLst+') values (' rqteInsert = beginRqteInsert+bindVar+')' psmt = ConTrg.prepareStatement(rqteInsert) # Statement source + metadata readSrc = ConSrc.createStatement() sqlSrc="""select <%=odiRef.getPop("DISTINCT_ROWS")%> <%=odiRef.getColList("", "[EXPRESSION]", ",\n\t", "", "(((INS or UPD) and !TRG) and REW)")%> , <% if (odiRef.getPop("HAS_JRN").equals("0")) { %> 'I' <%=odiRef.getInfo("SRC_COL_ALIAS_WORD")%> IND_UPDATE <%} else {%> JRN_FLAG <%}%> from <%=odiRef.getFrom()%> where (1=1) <%=odiRef.getJoin()%> <%=odiRef.getFilter()%> <%=odiRef.getJrnFilter()%> <%=odiRef.getGrpBy()%> <%=odiRef.getHaving()%>""" readSrc.setFetchSize(100) logger.record_time('Select action on source', LOG_ACTION) logger.record_time(sqlSrc, LOG_ACTION) rqteSrc = readSrc.executeQuery(sqlSrc) metaDataSrc = rqteSrc.getMetaData() # Set statement nb = None record = '' nbLine = 0 nbErrors = 0 psmt.clearBatch() while rqteSrc.next(): resLine='' record='' for nb in range (nbCol+1): colType=metaDataSrc.getColumnType(nb+1) if colType == sql.Types.VARCHAR: #Code 12 resline=rqteSrc.getString(nb+1) psmt.setString(nb+1,resline) elif colType == sql.Types.CHAR: #Code 1 resline=rqteSrc.getString(nb+1) psmt.setString(nb+1,resline) elif colType == sql.Types.INTEGER: #Code 4 resline=rqteSrc.getInt(nb+1) psmt.setInt(nb+1,resline) elif colType == sql.Types.NUMERIC: #Code 2 resline=rqteSrc.getBigDecimal(nb+1) psmt.setBigDecimal(nb+1,resline) elif colType == sql.Types.DECIMAL: #Code 3 resline=rqteSrc.getBigDecimal(nb+1) psmt.setBigDecimal(nb+1,resline) elif colType == sql.Types.DATE: #Code 91 resline=rqteSrc.getDate(nb+1) psmt.setDate(nb+1,resline) elif colType == sql.Types.TIME: #Code 92 resline=rqteSrc.getTime(nb+1) psmt.setTime(nb+1,resline) elif colType == sql.Types.TIMESTAMP: #Code 93 resline=rqteSrc.getTimestamp(nb+1) psmt.setTimestamp(nb+1,resline) elif colType == sql.Types.BIGINT: #Code -5 resline=rqteSrc.getLong(nb+1) psmt.setLong(nb+1,resline) elif colType == sql.Types.BINARY: #Code -2 resline=rqteSrc.getBytes(nb+1) psmt.setBytes(nb+1,resline) elif colType == sql.Types.BIT: #Code -7 resline=rqteSrc.getBoolean(nb+1) psmt.setBoolean(nb+1,resline) elif colType == sql.Types.DOUBLE: #Code 8 resline=rqteSrc.getDouble(nb+1) psmt.setDouble(nb+1,resline) elif colType == sql.Types.LONGVARBINARY: #Code -4 resline=rqteSrc.getBytes(nb+1) psmt.setBytes(nb+1,resline) elif colType == sql.Types.LONGVARCHAR: #Code -1 resline= rqteSrc.getString(nb+1) psmt.setString(nb+1,resline) elif colType == sql.Types.OTHER: #Code 1111 resline=rqteSrc.getObject(nb+1) psmt.setObject(nb+1,resline) elif colType == sql.Types.REAL: #Code 7 resline=rqteSrc.getFloat(nb+1) psmt.setFloat(nb+1,resline) elif colType == sql.Types.SMALLINT: #Code 5 resline=rqteSrc.getShort(nb+1) psmt.setShort(nb+1,resline) elif colType == sql.Types.TINYINT: #Code -6 resline=rqteSrc.getByte(nb+1) psmt.setByte(nb+1,resline) elif colType == sql.Types.VARBINARY: #Code -3 resline=rqteSrc.getBytes(nb+1) psmt.setBytes(nb+1,resline) record=record+str(resline)+', ' resline = '' try: logger.record_raw(str(nbLine) +' : ' + beginRqteInsert+' '+record[:-2]+' )' , LOG_ALL) psmt.executeUpdate() except: logger.record_raw(str(nbLine) +' : ' + beginRqteInsert+' '+record[:-2]+' )' , LOG_ERROR) nbErrors+=1 if nbErrors >= maxErrors: raise "Too much errors in the Load step", "The error limit has been reached. A maximum of "+str(maxErrors)+" errors was allowed." nbLine=nbLine+1 finally: print 'OK' ]]> null null null null null null null null null null null null null null null null null null null ( <%=odiRef.getColList("", "[COL_NAME]", ",\n\t", "", "(((INS or UPD) and !TRG) and REW)")%>, IND_UPDATE ) select <%=odiRef.getColList("", "E.[COL_NAME]", ",\n\t", "", "(((INS or UPD) and !TRG) and REW)")%>, 'I' IND_UPDATE from <%=odiRef.getTable("L","ERR_NAME", "W")%> <%=odiRef.getInfo("DEST_TAB_ALIAS_WORD")%> E where not exists ( select 'X' from <%=odiRef.getTable("L","INT_NAME","A")%> <%=odiRef.getInfo("DEST_TAB_ALIAS_WORD")%> T where <%=odiRef.getColList("", "E.[COL_NAME]\t= T.[COL_NAME]", "\n\t\tand\t", "", "(((INS or UPD) and !TRG) and REW)")%> ) and E.ORIGIN = '<%=odiRef.getInfo("CT_ORIGIN")%>' and E.ERR_TYPE = '<%=odiRef.getInfo("CT_ERR_TYPE")%>' ]]> null null null null null null null null null null null null null null null null null null null null where exists ( select 'X' from <%=odiRef.getTable("L","INT_NAME","A")%> <%=odiRef.getInfo("DEST_TAB_ALIAS_WORD")%> I where <%=odiRef.getColList("", odiRef.getTable("L","TARG_NAME","A") + ".[COL_NAME]\t= I.[COL_NAME]", "\n\t\tand \t", "", "(UK)")%> and IND_UPDATE = 'D' ) ]]> null null null null null null null null null null null null null null null null null null null null null where IND_UPDATE = 'D' ]]> null null null null null null null null null null null null null null null null null null null ]]> null null null null null null null null null null null null null null null null null null null set IND_UPDATE = 'U' where exists ( select 'X' from <%=odiRef.getTable("L", "TARG_NAME", "A")%> <%=odiRef.getInfo("DEST_TAB_ALIAS_WORD")%> T where <%=odiRef.getColList("", snpRef.getTable("L", "INT_NAME", "A")+".[COL_NAME]\t= T.[COL_NAME]", "\n\t\tand\t", "", "(UK)")%> ) ]]> null null null null