YUGONG ORACLE 迁移 MYSQL (多表合并单张表迁移)
最近,通过研究配置文件到实现,花费时间蛮久得,大概基本了解了配置文件配置..
整个合并到单表迁移流程:
查找到主连接表Mrvin_join >>> 按配置要求查询特定得需求值指定合并表SCOTT.join_name >>> 自定义数据类型转换名表名(简单理解就是变相把主连接扩展了字段以及数据),所以,目标端得表结构以及名字,应该设置成主连接表名marvin.marvin_join,否则可能报错..(这里是我对配置文件得分解,若有错误,请指正..谢谢)
再简化一句话:就是目标端除下需要增加得字段,其他所有字段包括表名都需要跟主连接表(marvin_join)一样
PS: 当时卡在报错MARVIN.MARVIN_JOIN表找不到大概两天....一直排查源端,其实源端正常...表存在、自定义数据转换正常、yugong权限GURD以及DBA正常、结果发现是由于MYSQL大小写敏感问题.....目标端应用数据找不到表...真醉了,修改配置文件my.cnf,增加参数lower_case_table_names=1,忽略表大小写即可解决
YUGONG前期其他准备工作,参考文章链接:www.cndba.cn/Marvinn/article/2737,这里步骤直接跳过,直入主题开始参数配置文件,以及DataTranslator得配置,以及源端与目标端的表的创建
注意:当前环境之前的单表对单表是仍然存在且在运行的,我并没有去除,也正好可以看看,各种需求放一起的配置
参数文件配置,沿用之前的配置,只需更改参数白名单即可。至于MYSQL数据库的选择,自行定
yugong.database.source.username=yugong
yugong.database.source.password=yugong
yugong.database.source.type=ORACLE
yugong.database.source.url=jdbc:oracle:thin:@//172.41.176.122:1521/orcl
yugong.database.source.encode=UTF-8
yugong.database.source.poolSize=30
yugong.database.target.url=jdbc:mysql://172.41.176.114:3306/test
yugong.database.target.url=jdbc:mysql://172.41.176.114:3306/marvin 可以指定一张表对应一个数据库mysql数据库
yugong.database.target.username=yugong
yugong.database.target.password=yugong
yugong.database.target.type=MYSQL
yugong.database.target.encode=UTF-8
yugong.database.target.poolSize=30
yugong.table.white=marvin.oracle1,scott.oracle2,marvin.marvin_join,scott.join_name --迁移表
yugong.table.mode=ALL
yugong.extractor.dump=true --这里DUMP出捕获日志,以便查看源端是否正常捕获数据,默认是false
yugong.applier.dump=true --这里DUMP出应用日志,以便查看目标端是否应用成功,默认false
新建DataTranslator文件(自定义数据转换)
注意:多表合并变单表,DataTranslator命名是根据主连接名来定夺,即当前主连接表是Marvin.marvin_join,被连接的是Scott.join_name,
所以,这里命名为MarvinJoinDataTranslator.java,另外,mysql对应表名也应该为marvin_join,应用日志会报错找不到表,(这里是按我的我理解,如有其他问题,可以在下留言,谢谢),当然,后续迁移完,表名、字段名、数据类型更改都是可以的…但是,如果想OGG一样,做数据同步,就不能更改了(有能力改改源码,应该也是可以实现的)
[yugong@rac2 translator]$ cp YugongExampleJoinDataTranslator.java MarvinJoinDataTranslator.java
这里需copy成主连接的表的名字MarvinJoinDataTranslator,至于为什么,参考之前文章 单表对单表迁移文章
根据模版文件配置下哈,这里全部去掉示例,以便真实了解如何配置
[yugong@rac2 translator]$ cat MarvinJoinDataTranslator.java
package com.taobao.yugong.translator;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Types;
import javax.sql.DataSource;
import org.springframework.dao.DataAccessException;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.ResultSetExtractor;
import com.taobao.yugong.common.db.meta.ColumnMeta;
import com.taobao.yugong.common.db.meta.ColumnValue;
import com.taobao.yugong.common.model.record.Record;
public class MarvinJoinDataTranslator extends BackTableDataTranslator implements DataTranslator {
// 更改Java类的名字MarvinJoinDataTranslator
public boolean translator(DataSource sourceDs, Record record) {
ColumnValue idColum = record.getColumnByName("md"); //这里根据主连接表Marvin_join确定
if (idColum != null) {
// 需要根据marvin_join.md字段,和另一张表做join,提取到关联md记录的表中mname字段,合并输出到一个目标表
//注意:若两表不再同一SCHEMA,就需要当前迁移用户,如配置文件中yugong用户有足够权限访问其他SCHEMA中的表(当前是DBA权限,否则查询会报错)
// 下面SELECT语句,相关可更改的信息,1、提取到关联md记录的JOIN_NAME表中MNAME字段,2、跟主连接表Marvin_join.md做匹配的表JOIN_MD字段,这两个决定了
// 另外,可以自己增加合并表字段的值
JdbcTemplate jdbcTemplate = new JdbcTemplate(sourceDs);
String name_value = (String) jdbcTemplate.query("select MNAME FROM SCOTT.JOIN_NAME WHERE JOIN_MD = "
+ idColum.getValue().toString(), new ResultSetExtractor() {
public Object extractData(ResultSet rs) throws SQLException, DataAccessException {
if (rs.next()) {
return rs.getString("MNAME");
}
return null;
}
});
ColumnMeta nameMeta = new ColumnMeta("mname", Types.VARCHAR); //该mname名是上面的被连接表JON_NAME的字段名
ColumnValue nameColumn = new ColumnValue(nameMeta, name_value);
record.addColumn(nameColumn);
}
return super.translator(record);
}
}
根据上面配置要求以及说明,创建测试表数据,其实正常逻辑是根绝表字段配置相关参数配置文件,但我这里出于搞清实际到底怎么配置,进行了逻辑反推,来验证….,验证成功,以后就可以根据我对配置文件DataTranslator的注释,进行需求配置环境了..
创建测试表数据
源端:
create table marvin.marvin_join
(
md NUMBER(11) ,
alias_name char(32) default ' ' not null,
CONSTRAINT marvin_join_pk_id PRIMARY KEY (id)
);
create table scott.join_name
(
id NUMBER(11) ,
mname char(32) default ' ' not null,
join_md NUMBER(11) ,
CONSTRAINT join_name_join_pk_id PRIMARY KEY (id)
);
源端两张表为1:1模型,通过marvin_join.md = join_name.join_md进行关联
源端插入测试数据:
declare
i number;
begin
for i in 1..10000 loop
insert into marvin.marvin_join values(i,'marvinn');
end loop;
commit;
end;
/
decalre
i number
begin
for i in 1..10000 loop
insert into scott.join_name values(i,'marvin',i);
end loop;
commit;
end;
/
目标端:
create table marvin.marvin_join
(
id bigint(20) unsigned auto_increment,
name varchar(32) ,
alias_name varchar(32),
CONSTRAINT marvin_join_name_pk_id PRIMARY KEY (id)
);
注意:经查看自定义数据文件DataTranslator配置文件,发现,两边合并流程为:
查找到主连接表Mrvin_join >>> 按配置要求查询特定得需求值指定合并表SCOTT.join_name >>> 自定义数据类型转换名表名(简单理解就是变相把主连接扩展了字段以及数据),所以,目标端得表结构以及名字,应该设置成主连接表名marvin.marvin_join,否则可能报错..(这里是我对配置文件理解,若有错误,请指正..谢谢)
再简化一句话:就是目标端除下需要增加得字段,其他所有字段包括表名都需要跟主连接表(marvin_join)一样
重新启动
yugong@rac2:/yugong/bin>sh stop.sh
yugong@rac2:/yugong/bin>sh start.sh
查看全局日志:
[yugong@rac2 yugong]$ tail -f table.log
发现之前的单表对单表正常,刚配的多表对单表异常
2018-04-19 17:30:22.219 [pool-2-thread-2] INFO com.taobao.yugong.common.stats.ProgressTracer - {未启动:0,全量中:0,增量中:0,已追上:2,异常数:2}
2018-04-19 17:30:22.220 [pool-2-thread-2] INFO com.taobao.yugong.common.stats.ProgressTracer - 异常数:[SCOTT.JOIN_NAME, MARVIN.MARVIN_JOIN]
2018-04-19 17:30:22.220 [pool-2-thread-2] INFO com.taobao.yugong.common.stats.ProgressTracer - 已完成:[MARVIN.ORACLE1, SCOTT.ORACLE2]
2018-04-19 17:31:22.220 [pool-2-thread-2] INFO com.taobao.yugong.common.stats.ProgressTracer - {未启动:0,全量中:0,增量中:0,已追上:2,异常数:2}
2018-04-19 17:31:22.220 [pool-2-thread-2] INFO com.taobao.yugong.common.stats.ProgressTracer - 异常数:[SCOTT.JOIN_NAME, MARVIN.MARVIN_JOIN]
2018-04-19 17:31:22.220 [pool-2-thread-2] INFO com.taobao.yugong.common.stats.ProgressTracer - 已完成:[MARVIN.ORACLE1, SCOTT.ORACLE2]
检查特定表marvin.marvin_join 捕获日志以及表日志
[root@rac2 MARVIN.MARVIN_JOIN]# pwd
/yugong/logs/MARVIN.MARVIN_JOIN
- Schema: MARVIN , Table: MARVIN_JOIN , Type: I
-----------------
---Pks
ColumnValue [column=ColumnMeta[name=MD,type=3], value=999]
---Columns
ColumnValue [column=ColumnMeta[name=ALIAS_NAME,type=1], value=marvinn ]
---END
-----------------
- Schema: MARVIN , Table: MARVIN_JOIN , Type: I
-----------------
---Pks
ColumnValue [column=ColumnMeta[name=MD,type=3], value=1000]
---Columns
ColumnValue [column=ColumnMeta[name=ALIAS_NAME,type=1], value=marvinn ]
---END
****************************************************
[root@rac2 MARVIN.MARVIN_JOIN]# tail -5 extractor.log
ColumnValue [column=ColumnMeta[name=MD,type=3], value=1000]
---Columns
ColumnValue [column=ColumnMeta[name=ALIAS_NAME,type=1], value=marvinn ]
---END
****************************************************
Table.log显示:表MARVIN.MARVIN_JOIN未发现
Caused by: com.taobao.yugong.exception.YuGongException: table[MARVIN.MARVIN_JOIN] is not found
at com.taobao.yugong.common.db.meta.TableMetaGenerator$1.doInConnection(TableMetaGenerator.java:75)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:342)
at com.taobao.yugong.common.db.meta.TableMetaGenerator.getTableMeta(TableMetaGenerator.java:51)
at com.taobao.yugong.applier.FullRecordApplier.getSqlUnit(FullRecordApplier.java:197)
at com.taobao.yugong.applier.FullRecordApplier.doApply(FullRecordApplier.java:87)
at com.taobao.yugong.applier.MultiThreadFullRecordApplier$1.run(MultiThreadFullRecordApplier.java:93)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
检查特定表以及scott.join_name日志
[root@rac2 SCOTT.JOIN_NAME]# tail -20f extractor.log
---END
-----------------
- Schema: SCOTT , Table: JOIN_NAME , Type: I
-----------------
---Pks
ColumnValue [column=ColumnMeta[name=ID,type=3], value=999]
---Columns
ColumnValue [column=ColumnMeta[name=JOIN_MD,type=3], value=999]
ColumnValue [column=ColumnMeta[name=MNAME,type=1], value=marvin ]
---END
-----------------
- Schema: SCOTT , Table: JOIN_NAME , Type: I
-----------------
---Pks
ColumnValue [column=ColumnMeta[name=ID,type=3], value=1000]
---Columns
ColumnValue [column=ColumnMeta[name=JOIN_MD,type=3], value=1000]
ColumnValue [column=ColumnMeta[name=MNAME,type=1], value=marvin ]
---END
****************************************************
Table.log显示:表SCOTT.JOIN_NAME未发现
Caused by: com.taobao.yugong.exception.YuGongException: table[SCOTT.JOIN_NAME] is not found
at com.taobao.yugong.common.db.meta.TableMetaGenerator$1.doInConnection(TableMetaGenerator.java:75)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:342)
at com.taobao.yugong.common.db.meta.TableMetaGenerator.getTableMeta(TableMetaGenerator.java:51)
at com.taobao.yugong.applier.FullRecordApplier.getSqlUnit(FullRecordApplier.java:197)
at com.taobao.yugong.applier.FullRecordApplier.doApply(FullRecordApplier.java:87)
at com.taobao.yugong.applier.MultiThreadFullRecordApplier$1.run(MultiThreadFullRecordApplier.java:93)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
PS: 该问题惆怅了好久....一直排查源端未果...才发现解决....真是尴尬
发现,Table.log显示两张表都未发现,排查源端表是都是存在得以及自定义数据转换DataTranslator名也未存在错误,而且源端两个表数据都捕获成功,说明源端并未存在问题,接着排查目标端.表存在....但是MYSQL对表名大小写敏感...尝试修改my.cnf文件增加到mysqld下忽略大小写配置,
[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
skip-external-locking
key_buffer_size = 16K
max_allowed_packet = 1M
table_open_cache = 4
sort_buffer_size = 64K
read_buffer_size = 256K
read_rnd_buffer_size = 256K
net_buffer_length = 2K
#thread_stack = 128K
lower_case_table_names=1
重启mysql以及yugong,继续查看特定表Table.log.发现MARVIN.MARVIN_JOIN表Table.log发生变化为 :
Caused by: java.util.concurrent.ExecutionException: com.taobao.yugong.exception.YuGongException: not found column[MD] in target db
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.taobao.yugong.common.utils.thread.ExecutorTemplate.waitForResult(ExecutorTemplate.java:77)
... 5 more
Caused by: com.taobao.yugong.exception.YuGongException: not found column[MD] in target db
at com.taobao.yugong.applier.AbstractRecordApplier.checkColumns(AbstractRecordApplier.java:56)
at com.taobao.yugong.applier.FullRecordApplier.getSqlUnit(FullRecordApplier.java:249)
at com.taobao.yugong.applier.FullRecordApplier.doApply(FullRecordApplier.java:87)
at com.taobao.yugong.applier.MultiThreadFullRecordApplier$1.run(MultiThreadFullRecordApplier.java:93)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
报错日志为: not found column[MD] in target db ,再次验证 创建测试表数据中得 注意 那段话...
查看目标端表结构:
mysql> desc marvin.marvin_join;
+------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------------------+------+-----+---------+----------------+
| id | bigint(20) unsigned | NO | PRI | NULL | auto_increment |
| name | varchar(32) | YES | | NULL | |
| alias_name | varchar(32) | YES | | NULL | |
+------------+---------------------+------+-----+---------+----------------+
3 rows in set (0.00 sec)
查看源端marvin_join:
SQL> desc MARVIN.MARVIN_JOIN
Name Null? Type
----------------------------------------- -------- ----------------------------
MD NOT NULL NUMBER(11)
ALIAS_NAME NOT NULL CHAR(32)
发现目标端确实不同...root用户登录修改目标表端表ID字段为MD,对了还有name字段,再合并得表中是并不存在得字段(即scott.join_name不存在该name字段),实际存在得是mname,所以也需要一并修改.
mysql> use marvin;
Database changed
mysql> alter table marvin_join change id md bigint(20) unsigned;
Query OK, 0 rows affected (0.07 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> alter table marvin_join change name mname varchar(32);
Query OK, 0 rows affected (0.04 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> desc marvin_join;
+------------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------------------+------+-----+---------+-------+
| md | bigint(20) unsigned | NO | PRI | 0 | |
| mname | varchar(32) | YES | | NULL | |
| alias_name | varchar(32) | YES | | NULL | |
+------------+---------------------+------+-----+---------+-------+
3 rows in set (0.00 sec)
重启yugong
[yugong@rac2 bin]$ sh stop.sh
rac2: stopping yugong 24961 ...
Oook! cost:1
[yugong@rac2 bin]$ sh startup.sh
cd to /yugong/bin for workaround relative path
LOG CONFIGURATION : /yugong/bin/../conf/logback.xml
yugong conf : /yugong/bin/../conf/yugong.properties
CLASSPATH :/yugong/bin/../conf:/yugong/bin/../lib/yugong-1.0.3.jar:/yugong/bin/../lib/spring-2.5.6.jar:/yugong/bin/../lib/slf4j-api-1.6.0.jar:/yugong/bin/../lib/oro-2.0.8.jar:/yugong/bin/../lib/ojdbc14-10.2.0.3.0.jar:/yugong/bin/../lib/mysql-connector-java-5.1.35.jar:/yugong/bin/../lib/mail-1.4.7.jar:/yugong/bin/../lib/logback-core-1.0.6.jar:/yugong/bin/../lib/logback-classic-1.0.6.jar:/yugong/bin/../lib/jcl-over-slf4j-1.6.0.jar:/yugong/bin/../lib/guava-19.0.jar:/yugong/bin/../lib/fastjson-1.2.8.jar:/yugong/bin/../lib/druid-1.0.17.jar:/yugong/bin/../lib/commons-logging-1.1.1.jar:/yugong/bin/../lib/commons-lang-2.6.jar:/yugong/bin/../lib/commons-io-2.4.jar:/yugong/bin/../lib/commons-configuration-1.9.jar:/yugong/bin/../lib/activation-1.1.jar:.:/usr/lib/jvm/java-1.8.0-openjdk/lib:/usr/lib/jvm/java-1.8.0-openjdk/jre/lib
cd to /yugong/bin for continue
查看全局Table.log
2018-04-23 11:37:27.249 [YuGongInstance-SCOTT.JOIN_NAME] INFO com.taobao.yugong.controller.YuGongInstance - table[SCOTT.JOIN_NAME] is error , current status:NORMAL !
2018-04-23 11:38:24.475 [pool-2-thread-2] INFO com.taobao.yugong.common.stats.ProgressTracer - {未启动:0,全量中:0,增量中:0,已追上:3,异常数:1}
2018-04-23 11:38:24.476 [pool-2-thread-2] INFO com.taobao.yugong.common.stats.ProgressTracer - 异常数:[SCOTT.JOIN_NAME]
发现[SCOTT.JOIN_NAME]表异常可以忽略,本身是被驱动表...即使你现在查看特定表SCOTT.JOIN_NAME日志Table,还是报表[SCOTT.JOIN_NAME],这是因为目标端端根本不存在该表...只需要捕获进程日志正常,捕获到数据即可...
查看MARVIN.MARVIN_JOIN表Table.log:
2018-04-23 11:37:30.878 [MultiThreadFullRecordApplier-MARVIN.MARVIN_JOIN] WARN com.alibaba.druid.pool.DruidDataSource - not full timeout retry : 1
2018-04-23 11:37:30.878 [MultiThreadFullRecordApplier-MARVIN.MARVIN_JOIN] WARN com.alibaba.druid.pool.DruidDataSource - not full timeout retry : 1
2018-04-23 11:37:39.791 [YuGongInstance-MARVIN.MARVIN_JOIN] INFO com.taobao.yugong.common.stats.StatAggregation - {总记录数:5000,采样记录数:5000,同步TPS:329,最长时间:10835,最小时间:1017,平均时间:3035}
2018-04-23 11:37:44.062 [YuGongInstance-MARVIN.MARVIN_JOIN] INFO com.taobao.yugong.common.stats.StatAggregation - {总记录数:10000,采样记录数:5000,同步TPS:1170,最长时间:906,最小时间:785,平均时间:854}
2018-04-23 11:37:44.062 [YuGongInstance-MARVIN.MARVIN_JOIN] INFO c.t.yugong.extractor.oracle.OracleAllRecordExtractor - table [MARVIN.MARVIN_JOIN] full extractor is end , next auto start inc extractor
2018-04-23 11:37:45.473 [YuGongInstance-MARVIN.MARVIN_JOIN] INFO c.t.y.e.oracle.OracleMaterializedIncRecordExtractor - table[MARVIN.MARVIN_JOIN] now is NO_UPDATE ...
2018-04-23 11:37:46.474 [YuGongInstance-MARVIN.MARVIN_JOIN] INFO c.t.y.e.oracle.OracleMaterializedIncRecordExtractor - table[MARVIN.MARVIN_JOIN] now is NO_UPDATE ...
发现数据已经被同步到了MYSQL目标端.....
验证数据
其实验证数据,主要是验证合并过来得SCOTT.JOIN_NAME字段mname数据是否存在...
源端:
SQL> select count(*) from marvin.marvin_join;
COUNT(*)
----------
10000
SQL> select count(*) from scott.join_name;
COUNT(*)
----------
10000
SQL> select mname from scott.join_name where rownum<=10;
MNAME
--------------------------------
marvin
marvin
marvin
marvin
marvin
marvin
marvin
marvin
marvin
marvin
10 rows selected.
目标端MYSQL:
mysql> desc marvin_join;
+------------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------------------+------+-----+---------+-------+
| md | bigint(20) unsigned | NO | PRI | 0 | |
| mname | varchar(32) | YES | | NULL | |
| alias_name | varchar(32) | YES | | NULL | |
+------------+---------------------+------+-----+---------+-------+
3 rows in set (0.00 sec)
mysql> select count(*) from marvin_join;
+----------+
| count(*) |
+----------+
| 10000 |
+----------+
1 row in set (0.00 sec)
mysql> select * from marvin_join limit 10;
+----+----------------------------------+----------------------------------+
| md | mname | alias_name |
+----+----------------------------------+----------------------------------+
| 1 | marvin | marvinn |
| 2 | marvin | marvinn |
| 3 | marvin | marvinn |
| 4 | marvin | marvinn |
| 5 | marvin | marvinn |
| 6 | marvin | marvinn |
| 7 | marvin | marvinn |
| 8 | marvin | marvinn |
| 9 | marvin | marvinn |
| 10 | marvin | marvinn |
+----+----------------------------------+----------------------------------+
10 rows in set (0.00 sec)
可以看到,数据字段mname确实被合并到了表marin_join中,数据显示正常....
至此…yugong ORACLE —>>> MYSQL 多张表合并一张表迁移成功….
版权声明:本文为博主原创文章,未经博主允许不得转载。



