canal, 是阿里巴巴出品的基于MySQL的数据增量日志解析, 提供数据订阅和消费的工具.
原理是实现MySQL的dump协议, 作为一个slave客户端, 向master发送dump请求. master收到请求后会传送相应的数据, 接着 canal 接收并解析binlog. 使用canal可以完成很多重要的业务功能, 比如变更通知/缓存更新/索引同步等等. 事不宜迟, 先体验一下.

下载并安装

mysql, version: 5.7.9, 安装略.
canal, version: v1.0.22

mysql 部分的配置

启用 bin log

sudo vim /etc/mysql/my.cnf

在[mysqld]的配置节下，插入：

log-bin=mysql-bin
binlog-format=ROW
server-id=1        #配置mysql replaction需要定义，不能和canal的slaveId重复

检查是否开启

show variables like 'log_%';

log_bin	ON
log_bin_basename	/var/lib/mysql/mysql-bin
log_bin_index	/var/lib/mysql/mysql-bin.index
log_bin_trust_function_creators	OFF
log_bin_use_v1_row_events	OFF
log_error	/var/log/mysqld.log
log_output	FILE
log_queries_not_using_indexes	OFF
log_slave_updates	OFF
log_slow_admin_statements	OFF
log_slow_slave_statements	OFF
log_throttle_queries_not_using_indexes	0
log_warnings	1

查询确认当前binlog模式:

SQL查询:
show variables like 'binlog_format';

canal 部分的配置

接下来是canal安装配置.

canal 的MySQL权限

canal的原理是模拟自己为mysql slave，所以这里一定需要作为 mysql slave 的相关权限.

创建canal用户及授权:

CREATE USER canal IDENTIFIED BY 'canal';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
FLUSH PRIVILEGES;
-- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;

构建 & 配置 canal

git clone git@github.com:alibaba/canal.git
cd canal
mvn clean install -Dmaven.test.skip=true -Denv=release
mkdir /tmp/canal
tar zxvf target/canal.deployer-1.0.22-SNAPSHOT.tar.gz -C /tmp/canal

编辑instance.properties :

cd /tmp/canal/
vi conf/example/instance.properties


#################################################
## mysql serverId
canal.instance.mysql.slaveId = 1234

# position info
canal.instance.master.address = 127.0.0.1:3306
canal.instance.master.journal.name =
canal.instance.master.position =
canal.instance.master.timestamp =

#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =

# username/password
canal.instance.dbUsername = canal
canal.instance.dbPassword = canal
canal.instance.defaultDatabaseName =
canal.instance.connectionCharset = UTF-8

# table regex
canal.instance.filter.regex = .*\\..*
# table black regex
canal.instance.filter.black.regex =

#################################################

怎么来填写这个配置, 首先先要查询master状态 :

mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000002 |      154 |              |                  |                   |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

如上, 把 mysql-bin.000002填入canal.instance.master.journal.name,
154 填入canal.instance.master.position, 即是:


...
canal.instance.master.journal.name = mysql-bin.000002
canal.instance.master.position = 154
....

启动 canal server

vim conf/canal.properties

暂时使用默认配置

sh bin/startup.sh

启动日志:
tail -f logs/canal/canal.log

具体instance的日志：
tail -f logs/example/example.log

sh bin/stop.sh

mysqlbinlog 测试

使用 mysqlbinlog 可以查看binlog, 例如:

mysqlbinlog -h127.0.0.1 -P33306 -ucanal -pcanal --read-from-remote-server -v --start-position=296 -d dbroute_01 mysql-bin.000003

使用 canal client 订阅数据

canal 客户端的代码程序太长, 这里就不贴了, 在github项目的example包下SimpleCanalClientTest.java

因为是使用ROW的binlog, 可以观察到变更的行和列, 而且哪些列更新了都可以获取, 日志示例:


================> binlog[mysql-bin.000001:65595295] , executeTime :  , delay : 1039ms
 BEGIN ----> Thread id: 131608
----------------> binlog[mysql-bin.000001:65595441] , name[test,city] , eventType : UPDATE , executeTime :  , delay : 1040ms
id : 56    type=int(11)
province_id : 5    type=int(11)
name : 包头市    type=varchar(50)    update=true
code : 229    type=varchar(50)
----------------
 END ----> transaction id: 269626487
================> binlog[mysql-bin.000001:65595556] , executeTime :  , delay : 1040ms

使用上面的example, 可以快速入门 canal API, 实现很多重要的业务功能.

参考

https://github.com/alibaba/canal/wiki/QuickStart
http://www.ilanni.com/?p=7816
http://m.blog.csdn.net/article/details?id=50824271