-
Notifications
You must be signed in to change notification settings - Fork 0
/
atom.xml
653 lines (441 loc) · 309 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>OMG_By</title>
<subtitle>沉心、静气、学习、总结、进步</subtitle>
<link href="/atom.xml" rel="self"/>
<link href="http://yoursite.com/"/>
<updated>2020-08-04T12:02:57.071Z</updated>
<id>http://yoursite.com/</id>
<author>
<name>OMG_By</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>Redis文章学习记录</title>
<link href="http://yoursite.com/2222/07/06/new/Redis/Redis%E6%96%87%E7%AB%A0%E5%AD%A6%E4%B9%A0%E8%AE%B0%E5%BD%95/"/>
<id>http://yoursite.com/2222/07/06/new/Redis/Redis文章学习记录/</id>
<published>2222-07-06T12:47:22.000Z</published>
<updated>2020-08-04T12:02:57.071Z</updated>
<content type="html"><![CDATA[<p>这篇博文主用来记录读到的Redis相关的文章</p><h2 id="实践案例"><a href="#实践案例" class="headerlink" title="实践案例"></a>实践案例</h2><ul><li><a href="https://mp.weixin.qq.com/s?__biz=MzAwMDU1MTE1OQ==&mid=2653547585&idx=1&sn=9a664b16f656f757632cd4eb29f9a5dc&scene=1&srcid=0726LGdy9zmneqsZyAekKukJ#rd" target="_blank" rel="noopener">近千节点的Redis Cluster高可用集群案例:优酷蓝鲸优化实战</a>：详细的介绍了Redis Cluster下心跳信息对带宽的影响。</li><li><a href="https://mp.weixin.qq.com/s/LeSSdT6bOEFzZyN26PRVzg" target="_blank" rel="noopener">干货 | 携程Redis海外机房数据同步实践</a>：携程开源 Redis 多数据中心复制管理系统 X-Pipe</li><li><a href="https://mp.weixin.qq.com/s?__biz=MzU1NDA4NjU2MA==&mid=2247486458&idx=1&sn=5d69ab965d84c69516caecbb776f0799&source=41#wechat_redirect" target="_blank" rel="noopener">优酷蓝鲸近千节点的Redis集群运维经验总结</a>：Redis Cluster运维时遇到的一些问题(<font color="red">实用</font>)</li><li><a href="https://mp.weixin.qq.com/s?__biz=MzAwMDU1MTE1OQ==&mid=2653547622&idx=1&sn=199cd6d8e3dff7c839935a7613d43e76&scene=1" target="_blank" rel="noopener">同程旅游缓存系统设计:如何打造Redis时代的完美体系</a>：携程Redis架构发展历程(<font color="red">推荐阅读</font>)</li><li><a href="https://mp.weixin.qq.com/s?__biz=MzAwMDU1MTE1OQ==&mid=2653547053&idx=1&sn=833fddbc83379d9cac8d7f757343412e" target="_blank" rel="noopener">Redis实战：如何构建类微博的亿级社交平台</a></li><li><a href="https://mp.weixin.qq.com/s?__biz=MzI4MTY5NTk4Ng==&mid=2247489693&idx=1&sn=b3138663029f41744203acf174725feb&source=41#wechat_redirect" target="_blank" rel="noopener">日请求量过亿，谈陌陌的Feed服务优化之路</a></li><li><a href="https://www.infoq.cn/article/2016/06/redis-storage-practise/" target="_blank" rel="noopener">Redis 应用实践：小红书海量 Redis 存储之道</a></li><li><a href="https://mp.weixin.qq.com/s?__biz=MzAwNTg2MDUyMw==&mid=2247483661&idx=1&sn=c924b3a2b098c4211b0044de180a1c0e" target="_blank" rel="noopener">如何实现高可用的redis集群</a>：客户端分片模式的Redis集群方案</li><li><a href="https://mp.weixin.qq.com/s?__biz=MzA4Nzg5Nzc5OA==&mid=2651660079&idx=1&sn=bca50ad39792deadf167077308120264" target="_blank" rel="noopener">唯品会大规模 Redis Cluster 的生产实践</a></li><li><a href="https://mp.weixin.qq.com/s?__biz=MzA3MzYwNjQ3NA==&mid=2651296671&idx=1&sn=366de50a6787963517ff6e096c9d1643" target="_blank" rel="noopener">这可能是最全的 Redis 集群方案介绍了</a></li><li><a href="https://mp.weixin.qq.com/s?__biz=MzAxMTEyOTQ5OQ==&mid=402004912&idx=1&sn=7517696a86f54262e60e1b5636d6cbe0" target="_blank" rel="noopener">Redis 集群的合纵与连横</a></li><li><a href="https://mp.weixin.qq.com/s?__biz=MzA5NzA0ODA1Nw==&mid=2650513422&idx=1&sn=846550678e98d0ea1adb444579ba558f" target="_blank" rel="noopener">Redis时延问题分析及应对</a></li><li><a href="https://developer.aliyun.com/article/46636#index_section" target="_blank" rel="noopener">干货来袭！Redis技术盛宴——阿里云Redis交流会火热召开</a></li><li><a href="https://mp.weixin.qq.com/s?__biz=MzAwMDU1MTE1OQ==&mid=2653547160&idx=1&sn=befd195e2aa788775aaf1cc3b6f6fab3" target="_blank" rel="noopener">首发丨360开源的类Redis存储系统:Pika</a></li><li><a href="https://redislabs.com/blog/the-endless-redis-replication-loop-what-why-and-how-to-solve-it/" target="_blank" rel="noopener">The Endless Redis Replication Loop: What, Why and How to Solve It</a></li><li><a href="https://redislabs.com/blog/top-redis-headaches-for-devops-replication-buffer/#.VxzASUC3PqB" target="_blank" rel="noopener">Top Redis Headaches for Devops – Replication Buffer</a></li><li><a href="https://mp.weixin.qq.com/s/g_OE7i66R9LqFgXmL9XvVg" target="_blank" rel="noopener">一次教科书级别的Redis高可用架构设计实践</a></li><li><a href="https://www.iteye.com/blog/carlosfu-2254566" target="_blank" rel="noopener">美团在Redis上踩过的一些坑-1.客户端周期性出现connect timeout</a></li><li><a href="https://mp.weixin.qq.com/s/l9A-z-Ol8FaSXg8Q0NU1UQ" target="_blank" rel="noopener">美团点评万亿级 KV 存储架构演进</a></li><li><a href="https://mp.weixin.qq.com/s/otae8hHeY-3Xb048ZSLJfQ" target="_blank" rel="noopener">牛逼！完美解密Redis与秒杀系统！！！</a></li></ul><ul><li><a href="https://mp.weixin.qq.com/s?__biz=MzAwODg3MDk0OQ==&mid=2247483698&idx=1&sn=9f82dcf764c86948a2ad5339d295cae4" target="_blank" rel="noopener">浅谈redis超时(二)</a>：翻译官方的redis延迟排查步骤。学习到了新命令(latency)</li><li><a href="https://www.iteye.com/blog/zsxxsz-2293332" target="_blank" rel="noopener">使用 redis_builder 管理 redis 集群</a>：C++编写的redis cluster集群工具</li><li><a href="http://soft.dog/2016/04/28/redis-docker-config/" target="_blank" rel="noopener">Redis 容器与配置</a></li><li><a href="https://mp.weixin.qq.com/s?__biz=MzI0NTE4NjA0OQ==&mid=2658351573&idx=2&sn=fb194491852b84b793233b2bebc456dc" target="_blank" rel="noopener">Docker化高可用redis集群</a>：docker搭建Redis主从+sentinel教程</li><li><a href="https://juejin.im/entry/579868e80a2b580061e352d2" target="_blank" rel="noopener">使用 Docker Compose 部署基于 Sentinel 的高可用 Redis 集群</a></li><li><a href="https://www.cnblogs.com/zhoujinyi/p/5585723.html" target="_blank" rel="noopener">Redis Sentinel 高可用实现说明</a> （要是早半个月看到这篇文章多好。自需求+正好遇到了文中提到的坑。PS：加一个坑：主从实例不能重命名config命令）</li><li><a href="https://mp.weixin.qq.com/s?__biz=MzU1NDA4NjU2MA==&mid=2247486425&idx=1&sn=c3aa5e3983b2320eecdda995b4fc81b6" target="_blank" rel="noopener">怎样打造一个分布式数据库 | 数据库功能深度解析</a></li><li><a href="https://blog.huangz.me/2020/redisraft.html" target="_blank" rel="noopener">使用 RedisRaft 构建强一致的 Redis 集群</a></li></ul><h2 id="机制、源码"><a href="#机制、源码" class="headerlink" title="机制、源码"></a>机制、源码</h2><ul><li><a href="https://mp.weixin.qq.com/s?__biz=MzAwNjQwNzU2NQ==&mid=2650342814&idx=1&sn=aba5c247a8c726cea3a86bac64d4a339" target="_blank" rel="noopener">Redis压缩列表原理与应用分析</a></li><li><a href="http://antirez.com/news/109" target="_blank" rel="noopener">Random notes on improving the Redis LRU algorithm</a></li><li><a href="https://mp.weixin.qq.com/s?__biz=MzI1MjExMDMzNQ==&mid=2651113608&idx=1&sn=0d1ec5f7e1201c2fdea72ab2df091cec" target="_blank" rel="noopener">Redis学习系列之八：Redis事务玩法</a>：Redis事务介绍和源码分析</li><li><a href="https://mp.weixin.qq.com/s?__biz=MzIwNzEzNDkxNQ==&mid=2650967934&idx=1&sn=1abd598b740ec3b42f76d59a14e4b788" target="_blank" rel="noopener">源码分析redis的sentinel在master宕机时是如何选择新的master的</a></li><li><a href="https://mp.weixin.qq.com/s?__biz=MzAwNTI4NzIxMQ==&mid=400500895&idx=1&sn=b27d22914d6b3cedc2fe16ccd02a1a01" target="_blank" rel="noopener">Redis主从同步失败案例的步步深入</a>：redis主从同步的一个bug(已修复)</li><li><a href="https://mp.weixin.qq.com/s?__biz=MzA3MzYwNjQ3NA==&mid=2651296996&idx=2&sn=5f4811d73e74e2a63b1cb0d3d532862a" target="_blank" rel="noopener">深入浅出 Redis Cluster 原理，来自一次干货分享</a>：Redis Cluster原理讲解</li><li><a href="https://www.jianshu.com/p/48b0d96740a5" target="_blank" rel="noopener">Redis和IO多路复用</a></li><li><a href="https://yq.aliyun.com/live/1153?spm=a2c4e.11154022.liveshowinfo.11.4022ebc42plKm5" target="_blank" rel="noopener">Redis主从复制演变过程</a> （很好的讲解了Redis各版本中复制的演变）</li><li><a href="https://mp.weixin.qq.com/s?__biz=MzA5ODM5MDU3MA==&mid=2650861715&idx=1&sn=e30a388f897c8953842fb5e8843048e4" target="_blank" rel="noopener">Redis 缓存失效机制</a></li><li><a href="http://zhangtielei.com/posts/blog-redis-quicklist.html" target="_blank" rel="noopener">Redis内部数据结构详解(5)——quicklist</a></li><li><a href="https://blog.csdn.net/Xiejingfa/article/details/51553370" target="_blank" rel="noopener">Redis源码剖析 - Redis持久化之RDB</a></li><li><a href="https://developer.aliyun.com/article/58703" target="_blank" rel="noopener">Redis源码学习——BIO</a></li><li><a href="https://developer.aliyun.com/article/62593" target="_blank" rel="noopener">Redis短连接性能优化</a></li><li><a href="https://hk.v2ex.com/t/646669" target="_blank" rel="noopener">Redis 多线程 IO 模型</a></li><li><a href="https://mp.weixin.qq.com/s?__biz=MzA4NTg1MjM0Mg==&mid=2657261425&idx=1&sn=d840079ea35875a8c8e02d9b3e44cf95&scene=0&key=c50f8b988e61749ab65cb6b9c037598ca0bafdf899895598f2904791c26b5e4283079646ce247da25d76658cd815b833&ascene=0&uin=MzM4Njg2NDU1" target="_blank" rel="noopener">Redis为什么用跳表而不用平衡树？</a></li><li><a href="https://developer.aliyun.com/article/62593" target="_blank" rel="noopener">Redis短连接性能优化</a></li><li><a href="https://mp.weixin.qq.com/s?__biz=MzIzNjUxMzk2NQ==&mid=2247483989&idx=1&sn=c1aac5b3eca9cb3f9e177d88ba189719" target="_blank" rel="noopener">深入浅出 Redis client/server交互流程</a></li></ul><h2 id="版本特性"><a href="#版本特性" class="headerlink" title="版本特性"></a>版本特性</h2><ul><li><a href="https://mp.weixin.qq.com/s/Gm1EzhbcZEWpw8eXmP_ibw" target="_blank" rel="noopener">Redis 6.0 新特性概览</a></li><li><a href="https://mp.weixin.qq.com/s/OGE2KivQzooNsWKJyvlgzQ" target="_blank" rel="noopener">客户端缓存功能</a> </li><li><a href="https://mp.weixin.qq.com/s/Mc9SpVeuRIe8Pi2HkPQWjQ" target="_blank" rel="noopener">带你100% 地了解 Redis 6.0 的客户端缓存</a></li><li><a href="https://www.infoq.cn/article/fe97cBWx6Hqk9qVvoW7r" target="_blank" rel="noopener">Redis 干货分享 | Redis 访问控制列表 (ACL)</a></li></ul><h2 id="Redis相关工具"><a href="#Redis相关工具" class="headerlink" title="Redis相关工具"></a>Redis相关工具</h2><ul><li><a href="https://developer.aliyun.com/article/691794" target="_blank" rel="noopener">数据迁移工具-redisSheke</a></li><li><a href="https://developer.aliyun.com/article/690463" target="_blank" rel="noopener">数据对比工具-redisFullCheck</a></li><li><a href="https://mp.weixin.qq.com/s?__biz=MjM5NzM0MjcyMQ==&mid=2650068463&idx=2&sn=d9aa4cc763a6b0554b031a4451fe09aa" target="_blank" rel="noopener">nredis-proxy 高性能Redis 服务中间件</a></li></ul><blockquote><p><a href="https://toutiao.io/tags/Redis" target="_blank" rel="noopener">https://toutiao.io/tags/Redis</a><br><a href="https://github.com/huangz1990/redis-3.0-annotated" target="_blank" rel="noopener">https://github.com/huangz1990/redis-3.0-annotated</a></p></blockquote>]]></content>
<summary type="html">
<p>这篇博文主用来记录读到的Redis相关的文章</p>
<h2 id="实践案例"><a href="#实践案例" class="headerlink" title="实践案例"></a>实践案例</h2><ul>
<li><a href="https://mp.weixin
</summary>
<category term="Redis" scheme="http://yoursite.com/categories/Redis/"/>
<category term="Redis" scheme="http://yoursite.com/tags/Redis/"/>
</entry>
<entry>
<title>MySQL文章学习记录</title>
<link href="http://yoursite.com/2222/07/06/new/MySQL/MySQL%E6%96%87%E7%AB%A0%E8%AE%B0%E5%BD%95/"/>
<id>http://yoursite.com/2222/07/06/new/MySQL/MySQL文章记录/</id>
<published>2222-07-06T12:47:22.000Z</published>
<updated>2020-08-04T11:25:13.103Z</updated>
<content type="html"><![CDATA[<p>这篇博文主用来记录读到的MySQL相关的文章</p><h2 id="机制系列"><a href="#机制系列" class="headerlink" title="机制系列"></a>机制系列</h2><ul><li><a href="https://jin-yang.github.io/post/mysql-protocol.html" target="_blank" rel="noopener">MySQL 通讯协议</a>：客户端与服务端连接过程中，密码怎样传输</li><li><a href="https://mp.weixin.qq.com/s/2jqPhy_6zefgrx0zqhgNCw" target="_blank" rel="noopener">活久见，为什么SHOW TABLE STATUS总是不更新</a></li><li><a href="https://blog.csdn.net/n88Lpo/article/details/81187372" target="_blank" rel="noopener">图解MySQL MySQL组提交(group commit)</a></li></ul><h2 id="故障及优化案例系列"><a href="#故障及优化案例系列" class="headerlink" title="故障及优化案例系列"></a>故障及优化案例系列</h2><ul><li><a href="https://mp.weixin.qq.com/s/VS534q8wv_rJxZ0EawhU9Q" target="_blank" rel="noopener">大规模多存储场景的数据库选型与服务平台建设</a></li><li><a href="https://mp.weixin.qq.com/s/hYsxeevggxwkObJyOqy3iA" target="_blank" rel="noopener">引号错位引起的故障</a> （我才不会告诉你踩过的一个坑：update xx set a=’x’ and b=’x’ where …）</li><li><a href="https://mp.weixin.qq.com/s/snJcudytuW4_BvgVk86IXw" target="_blank" rel="noopener">一个事务中调整语句顺序</a> （MySQL45讲中也说过：一个事务中，只有需要锁的时候才会去申请锁，事务提交时才会释放锁。所以将锁资源较多的语句放在后面执行。） </li><li><a href="https://mp.weixin.qq.com/s/wFath89f3xXDR1Nxd5g-mw" target="_blank" rel="noopener">库表字符集修改后，老字段仍为原来字符集</a> （修改库表字符集后，只会改变新增的字符集）</li><li><a href="https://mp.weixin.qq.com/s/jMU1lGhBwuIPrERqeUFAVQ" target="_blank" rel="noopener">聊聊 隐式转换</a> （真实遇到过其中的字符串转换成浮点型比较，查询结果不准确的问题。实测当长度为17位时会有问题）</li><li><a href="https://mp.weixin.qq.com/s/tWDQUPG5PYaeZilz7dd73Q" target="_blank" rel="noopener">一个特殊的隐式转换问题</a> （5.6版本IN中包括多个类型的话，会存在不能走索引的情况。<a href="http://mysql.taobao.org/monthly/2017/12/06/" target="_blank" rel="noopener">相关阿里月报</a>）</li><li><a href="https://mp.weixin.qq.com/s/f8ZQx3Iuz-BnJ05z8vN2GA" target="_blank" rel="noopener">Strace 解决性能问题案例一则</a> （问题排查工具，用事实说话，DB不背锅）</li><li><a href="https://mp.weixin.qq.com/s/MdfomhoKV4OBJ_jSv1d0Bg" target="_blank" rel="noopener">业务优化案例一则</a> (mysqlslap压测工具使用)</li><li><a href="https://mp.weixin.qq.com/s/2hFm_hZgbNwi8GsE5MFrQg" target="_blank" rel="noopener">MySQL 案例一则</a> （explicit_defaults_for_timestamp参数解释）<ul><li>explicit_defaults_for_timestamp</li></ul></li><li><a href="https://mp.weixin.qq.com/s/w4qtB5Dz_ybbiY-WRr72-Q" target="_blank" rel="noopener">再说 order by 优化</a> （<font color="red">很精髓的order by案例总结和优化建议。</font>）</li><li><a href="https://mp.weixin.qq.com/s/T6qeXpVoo-M9Iflsh-n1zA" target="_blank" rel="noopener">一次大量删除导致 MySQL 慢查的分析</a> （由于MVCC特性，如果存在长时间未提交的事务，会导致简单查询也可能会变得很慢）</li><li><a href="https://mp.weixin.qq.com/s/0y20YxeSvgwJM7ZUho1C7Q" target="_blank" rel="noopener">MySQL 大量sleeping before entering InnoDB 故障诊断</a> （很详细的一次问题排查步骤，里面有很多有用的命令）</li><li><a href="https://mp.weixin.qq.com/s/Edn_gPwcAHo5sYIzLJghzA" target="_blank" rel="noopener">哪些因素会导致慢查询？</a> （<font color="red">从应用到数据库，非常全面的列出了变慢的原因</font> 加一个：limit 时匹配不到数据会一直扫描。）</li><li><a href="https://mp.weixin.qq.com/s/AC9CwpSUTW6fXA-5mHxPGQ" target="_blank" rel="noopener">由MySQL复制延迟说起</a> （分析了复制延迟的主要原因和解决办法）</li><li><a href="https://mp.weixin.qq.com/s/safm3DQh2zYO9Pdk9ZXbkA" target="_blank" rel="noopener">为何COUNT很慢却不写SLOW LOG</a>：slow log记录规则</li><li><a href="https://mp.weixin.qq.com/s/7L-aBKmjORJ7Mg72Od47lw" target="_blank" rel="noopener">故障分析 | MySQL 优化案例 - select count(*)</a></li></ul><p>SQL变慢的原因主要有：</p><ol><li>SQL执行过程走索引不合理，导致执行缓慢。</li><li>使用合理索引，但是获取数据量比较多。(排序，临时表等)</li><li>网络重传丢包导致SQL变慢。</li><li>并发比较高的场景，请求排队处理，等待时间长。</li></ol><p>性能优化是一个老生常谈的问题，需要对相关流程和机制有很深入的研究才能对症下药。平时扩充、积累相关知识，在遇到问题后，才会有思路。经验很重要，但更多的是建立在平时的积累上。</p><h2 id="死锁系列"><a href="#死锁系列" class="headerlink" title="死锁系列"></a>死锁系列</h2><ul><li><a href="https://mp.weixin.qq.com/s/xUgzKPMZP777jdytrCL7oA" target="_blank" rel="noopener">漫谈死锁</a> （1. RC级别下也会存在Next-key 2. RC级别下，获取到不符合记录时会释放锁）<ul><li>innodb_print_all_deadlocks (将死锁信息记录到errorlog中参数)</li><li>innodb_status_output_locks (标准监控开关参数)</li></ul></li><li><a href="https://mp.weixin.qq.com/s/8XXimj2AIsRWj0fF-_iz3g" target="_blank" rel="noopener">如何阅读死锁日志</a> （锁组合）</li><li><a href="https://mp.weixin.qq.com/s/2GIrpSKHe3Y3KWMfDbA7jw" target="_blank" rel="noopener">MySQL 各种SQL语句加锁分析</a><ul><li>innodb_locks_unsafe_for_binlog</li></ul></li><li><a href="https://mp.weixin.qq.com/s/MywiDdffz13i8auSv4Xs9w" target="_blank" rel="noopener">死锁案例一</a> （简单插入唯一索引插入意向锁案例）</li><li><a href="https://mp.weixin.qq.com/s/7DehY-zQV1XHYufaII4oJg" target="_blank" rel="noopener">死锁案例二</a> （delete操作会有锁区间行为。PS：删除已有记录会导致锁范围变大)</li><li><a href="https://mp.weixin.qq.com/s/R1dJY4leZh5qRpUkvOsNBw" target="_blank" rel="noopener">死锁案例三</a> （insert同一区间时，唯一索引的插入意向锁导致的死锁）</li><li><a href="https://mp.weixin.qq.com/s/wXH9XFOK8wui7dDtR6Hzww" target="_blank" rel="noopener">死锁案例四</a> （并发插入时，插入意向锁不同阶段死锁。<font color="red">注意跟案例三的区别：本案例为X锁等待，案例三为S锁等待</font>)</li><li><a href="https://mp.weixin.qq.com/s/XCdLfyjSuWAEcWIjbcVfvg" target="_blank" rel="noopener">死锁案例五</a> （repalce into插入死锁情况）</li><li><a href="https://mp.weixin.qq.com/s/Lp7b2TGyFY-4uEN_M6Y8zg" target="_blank" rel="noopener">死锁案例六</a> （不同事务加锁顺序不一样，导致锁资源交叉影响导致的死锁)</li><li><a href="https://mp.weixin.qq.com/s/pC4XCpUOuNPMFj6UjuhWvw" target="_blank" rel="noopener">死锁案例七</a> （RC级别下，update不存在记录会持有lock_S + GAP锁）</li><li><a href="https://mp.weixin.qq.com/s/96CDhpgu5uUQ7qKYhKgt3w" target="_blank" rel="noopener">死锁案例八</a> （多并发插入下的又一插入意向锁竞争导致的死锁案例）</li><li><a href="https://mp.weixin.qq.com/s/h4Mwrowft2KKdTkqmRlYqw" target="_blank" rel="noopener">死锁案例九</a> （并发批量插入，导致的死锁）</li><li><a href="https://mp.weixin.qq.com/s/b9gNbdEHV3NNQrV9PKDPSw" target="_blank" rel="noopener">死锁案例十</a> （并发更新普通二级索引造成的死锁。PS：<font color="red">另一程度上说明索引不是随便加就行</font>）</li><li><a href="https://mp.weixin.qq.com/s/vMGlRtMQckJ1LUZaSBn5cw" target="_blank" rel="noopener">死锁案例十一</a> （<font color="red">5.6升级到5.7后，INSERT INTO .. ON DUPLICATE KEY语句锁模式加强</font>）</li><li><a href="https://mp.weixin.qq.com/s/SFdPDPhI270tnVI3uuOH0Q" target="_blank" rel="noopener">死锁案例十二</a> （唯一索引，多插入,插入意向锁死锁情况）</li><li><a href="!https://www.cnblogs.com/yulibostu/articles/9844061.html">一个最不可思议的死锁分析</a> （并发delete语句造成的死锁情况，里面引用了很多好文章，需要细品）</li></ul><p>分析死锁单从<code>show engines innodb status</code>中往往只能看到最后执行的语句，并不能很好的了解整个死锁形成过程。所以在分析死锁时，需要先了解两个死锁事务的具体逻辑和语句过程。<br>了解MySQL不同操作的加锁过程非常重要，这是对死锁形成过程分析的基础。</p><a id="more"></a><h2 id="MySQL8-0系列"><a href="#MySQL8-0系列" class="headerlink" title="MySQL8.0系列"></a>MySQL8.0系列</h2><p><img src="https://tva1.sinaimg.cn/large/007S8ZIlly1gglqmoolzdj311s0mg0wp.jpg" alt=""> <img src="https://tva1.sinaimg.cn/large/007S8ZIlly1gglqmwiccwj30iq0bita5.jpg" alt=""></p><ul><li><a href="https://mp.weixin.qq.com/s/-k99IVJ8tjFYZ4C1V_3nAQ" target="_blank" rel="noopener">MySQL8.0到目前为止有哪些真香特性</a></li><li><a href="https://opensource.actionsky.com/category/%e6%8a%80%e6%9c%af%e5%b9%b2%e8%b4%a7/mysql-new/" target="_blank" rel="noopener">8.0 新特性</a></li><li><a href="https://www.cndba.cn/Expect-le/article/3135" target="_blank" rel="noopener">MySQL8.0中已移除的特性，功能</a></li><li><a href="https://mp.weixin.qq.com/s/C6129OaeeLaodzLeRhWbvw" target="_blank" rel="noopener">MySQL 8.0 clone plugin 完整版</a>：官方支持的一种全新的数据库备份恢复方式。</li><li><a href="https://mp.weixin.qq.com/s/9GHIpT_YeUoZNJ2X6cS-YQ" target="_blank" rel="noopener">真香，MySQL 8.0.20，10倍性能提升</a>：doublewrite性能优化，由全局锁减小为实例锁</li><li><a href="https://opensource.actionsky.com/20191119-mysql/" target="_blank" rel="noopener">技术分享 | Hash join in MySQL 8</a></li><li><a href="https://mp.weixin.qq.com/s/gYledARCZHvrlnCEmh2mtw" target="_blank" rel="noopener">MySQL 8.0 hash join有重大缺陷？</a></li><li><a href="https://mp.weixin.qq.com/s/7FI87f6t3UvbE9GGhw8iVA" target="_blank" rel="noopener">一文读懂MySQL 8.0直方图</a>：不用加索引的优化方式，需要手动更新</li><li><a href="https://mp.weixin.qq.com/s/OYgWvJQr736rqpF1bsIJ-A" target="_blank" rel="noopener">MySQL 8.0中对EXISTS、NOT EXISTS的持续优化</a>：从8.0.16开始，增加对EXISTS的优化，和IN一样也支持自动转换成semi join</li><li><a href="https://opensource.actionsky.com/20200420-mysql/" target="_blank" rel="noopener">新特性解读 | 趋近完美的 Undo 空间</a></li><li><a href="https://opensource.actionsky.com/20200318-mysql/" target="_blank" rel="noopener">七个实验掌握 MySQL 8.0 角色功能</a></li><li><a href="https://opensource.actionsky.com/20190528-mysql8-index4/" target="_blank" rel="noopener">新特性解读 | MySQL 8.0 索引特性4-不可见索引</a></li><li><a href="https://opensource.actionsky.com/20190520-mysql8-0-index3/" target="_blank" rel="noopener">新特性解读 | MySQL 8.0 索引特性3 -倒序索引</a></li><li><a href="https://opensource.actionsky.com/20190507-mysql8-0-index2/" target="_blank" rel="noopener">新特性解读 | MySQL 8.0 索引特性2-索引跳跃扫描</a></li><li><a href="https://mp.weixin.qq.com/s/rQ7og6iXoOofFrnnz7vgbg" target="_blank" rel="noopener">MySQL 8.0 之Index Skip Scan</a></li><li><a href="https://blog.csdn.net/vkingnew/article/details/81267320" target="_blank" rel="noopener">MySQL 8.0.12 instant add column 体验，亿级数据秒速增加字段</a></li><li><p><a href="http://mysql.taobao.org/monthly/2018/11/09/" target="_blank" rel="noopener">MySQL · 最佳实践 · 8.0 CTE和窗口函数的用法</a></p></li><li><p><a href="https://opensource.actionsky.com/20200709-mysql/" target="_blank" rel="noopener">新特性解读 | MySQL 8.0 之原子 DDL</a></p></li><li><a href="https://opensource.actionsky.com/20200520-mysql/" target="_blank" rel="noopener">新特性解读 | binlog 压缩</a>：MySQL 从 8.0.20 开始集成 ZSTD 算法，推出binlog 压缩功能</li><li><a href="https://mp.weixin.qq.com/s/FDE31xEhl4hX6PidHChX0Q" target="_blank" rel="noopener">新特性解读 | 说说 MySQL 8 对于持久化变量的一些修改</a>：在命令行修改参数值的同时进行持久化，避免忘记修改配置文件导致实例重启后出现未知的问题。</li><li><a href="https://opensource.actionsky.com/20200325-mysql/" target="_blank" rel="noopener">新特性解读 | 8.0 新增 DML 语句（TABLE & VALUES）</a></li><li><a href="https://mp.weixin.qq.com/s/bNiNsP0tC4_t0ahSANU4wg" target="_blank" rel="noopener">新特性解读 | mysql 8.0 memcached api 新特性</a>：NOSQL机制集成的又一体现。使用telnet连接MySQL时遇到问题，可能原因看下面这篇文章</li><li><a href="https://www.cnblogs.com/yinzhengjie/p/10301516.html" target="_blank" rel="noopener">MySQL 8.0.14 新的密码认证方式和客户端链接</a>：新加密方式(caching_sha2_password)</li><li><a href="https://mp.weixin.qq.com/s/jfoH6D8NfoaNN7eexbzKGA" target="_blank" rel="noopener">MySQL 8.0密码认证机制升级，不知道可能导致业务不可用</a>：客户端与服务端加密方式不一样导致连接失败。</li><li><a href="https://mp.weixin.qq.com/s?__biz=MjM5NzAzMTY4NQ==&mid=2653933468&idx=1&sn=a6db84ebfedcf1c03a40d5351ae93be6&chksm=bd3b55f68a4cdce0de76de7abc2266bdb2bfd7e7cdf8696ffecd481c0ae550458ac82cff9a68&scene=158#rdZ" target="_blank" rel="noopener">解密MySQL 8.0 multi-valued indexes</a>：MySQL 8.0.17起，InnoDB引擎新增了对JSON数据类型的多值索引</li><li><a href="https://www.cndba.cn/Supreme_Aaron/article/4186" target="_blank" rel="noopener">MySQL 一个历史遗留“BUG”</a>：行锁多锁一条记录，MySQL45讲中提到过这个问题</li><li><a href="https://mp.weixin.qq.com/s/5xg-jy56u12Kq9BZjglULg" target="_blank" rel="noopener">MySQL 8.0 InnoDB压缩行格式性能测试</a>：新的压缩算法不会减少多少占用，在一定程度上影响了性能。需要根据业务场景谨慎选择</li><li><a href="https://mp.weixin.qq.com/s?__biz=MjM5NzAzMTY4NQ==&mid=2653933464&idx=1&sn=82bd2d6ddbd103efe46a51a357e872de&chksm=bd3b55f28a4cdce4b021b41457a188cbfa53544117de74cd7876b9ddd244f98e1bf9b4ab8be4&scene=158#rd" target="_blank" rel="noopener">MySQL 8.0.19客户端的一个小变化</a>：默认当查询到的数据有二进制数据的话，就会用十六进制方式展示出来</li></ul><h2 id="源码解析"><a href="#源码解析" class="headerlink" title="源码解析"></a>源码解析</h2><ul><li><a href="https://mp.weixin.qq.com/s/aR1fUQAkHdC0wAdJ5H6BHw" target="_blank" rel="noopener">主从替换之后的复制风暴</a> （复制风暴、seconds_behind_master计算方式）</li><li><a href="https://mp.weixin.qq.com/s/CjacOYs8gdkUPdnofktnXg" target="_blank" rel="noopener">show status和set gtid_mode 导致线程死锁案例</a> （<font color="red">TODO：gdb和通过源码分析问题思路实操</font>）</li></ul><h2 id="其他"><a href="#其他" class="headerlink" title="其他"></a>其他</h2><ul><li><a href="https://mp.weixin.qq.com/s/YAE_DidgZwHOzQ4NSZPyUg" target="_blank" rel="noopener">MySQL centos 6 vs 7的性能对比</a></li><li><font color="gray"><a href="https://mp.weixin.qq.com/s/HlAKday9K0D6AtCdg0uUtg" target="_blank" rel="noopener">xfs vs ext4 性能压测对比</a> (先mark，后面再细细品)</font></li><li><a href="https://mp.weixin.qq.com/s/bOdwi_zYOghBHX-eHDb5-w" target="_blank" rel="noopener">使用innobackup 2.4遇到的问题</a> （2.2是不能备份5.7 版本）</li><li><a href="https://mp.weixin.qq.com/s/ubiyT62zG3NGxTMSJIoY0g" target="_blank" rel="noopener">MySQL DBA如何利用strace/pstack/gdb来定位问题</a></li><li><a href="https://got.qq.com/webplat/info/news_version3/8616/8622/8625/8628/m7025/201407/271174.shtml" target="_blank" rel="noopener">MySQL在线加字段实现原理</a></li></ul>]]></content>
<summary type="html">
<p>这篇博文主用来记录读到的MySQL相关的文章</p>
<h2 id="机制系列"><a href="#机制系列" class="headerlink" title="机制系列"></a>机制系列</h2><ul>
<li><a href="https://jin-yang.github.io/post/mysql-protocol.html" target="_blank" rel="noopener">MySQL 通讯协议</a>：客户端与服务端连接过程中，密码怎样传输</li>
<li><a href="https://mp.weixin.qq.com/s/2jqPhy_6zefgrx0zqhgNCw" target="_blank" rel="noopener">活久见，为什么SHOW TABLE STATUS总是不更新</a></li>
<li><a href="https://blog.csdn.net/n88Lpo/article/details/81187372" target="_blank" rel="noopener">图解MySQL MySQL组提交(group commit)</a></li>
</ul>
<h2 id="故障及优化案例系列"><a href="#故障及优化案例系列" class="headerlink" title="故障及优化案例系列"></a>故障及优化案例系列</h2><ul>
<li><a href="https://mp.weixin.qq.com/s/VS534q8wv_rJxZ0EawhU9Q" target="_blank" rel="noopener">大规模多存储场景的数据库选型与服务平台建设</a></li>
<li><a href="https://mp.weixin.qq.com/s/hYsxeevggxwkObJyOqy3iA" target="_blank" rel="noopener">引号错位引起的故障</a> （我才不会告诉你踩过的一个坑：update xx set a=’x’ and b=’x’ where …）</li>
<li><a href="https://mp.weixin.qq.com/s/snJcudytuW4_BvgVk86IXw" target="_blank" rel="noopener">一个事务中调整语句顺序</a> （MySQL45讲中也说过：一个事务中，只有需要锁的时候才会去申请锁，事务提交时才会释放锁。所以将锁资源较多的语句放在后面执行。） </li>
<li><a href="https://mp.weixin.qq.com/s/wFath89f3xXDR1Nxd5g-mw" target="_blank" rel="noopener">库表字符集修改后，老字段仍为原来字符集</a> （修改库表字符集后，只会改变新增的字符集）</li>
<li><a href="https://mp.weixin.qq.com/s/jMU1lGhBwuIPrERqeUFAVQ" target="_blank" rel="noopener">聊聊 隐式转换</a> （真实遇到过其中的字符串转换成浮点型比较，查询结果不准确的问题。实测当长度为17位时会有问题）</li>
<li><a href="https://mp.weixin.qq.com/s/tWDQUPG5PYaeZilz7dd73Q" target="_blank" rel="noopener">一个特殊的隐式转换问题</a> （5.6版本IN中包括多个类型的话，会存在不能走索引的情况。<a href="http://mysql.taobao.org/monthly/2017/12/06/" target="_blank" rel="noopener">相关阿里月报</a>）</li>
<li><a href="https://mp.weixin.qq.com/s/f8ZQx3Iuz-BnJ05z8vN2GA" target="_blank" rel="noopener">Strace 解决性能问题案例一则</a> （问题排查工具，用事实说话，DB不背锅）</li>
<li><a href="https://mp.weixin.qq.com/s/MdfomhoKV4OBJ_jSv1d0Bg" target="_blank" rel="noopener">业务优化案例一则</a> (mysqlslap压测工具使用)</li>
<li><a href="https://mp.weixin.qq.com/s/2hFm_hZgbNwi8GsE5MFrQg" target="_blank" rel="noopener">MySQL 案例一则</a> （explicit_defaults_for_timestamp参数解释）<ul>
<li>explicit_defaults_for_timestamp</li>
</ul>
</li>
<li><a href="https://mp.weixin.qq.com/s/w4qtB5Dz_ybbiY-WRr72-Q" target="_blank" rel="noopener">再说 order by 优化</a> （<font color="red">很精髓的order by案例总结和优化建议。</font>）</li>
<li><a href="https://mp.weixin.qq.com/s/T6qeXpVoo-M9Iflsh-n1zA" target="_blank" rel="noopener">一次大量删除导致 MySQL 慢查的分析</a> （由于MVCC特性，如果存在长时间未提交的事务，会导致简单查询也可能会变得很慢）</li>
<li><a href="https://mp.weixin.qq.com/s/0y20YxeSvgwJM7ZUho1C7Q" target="_blank" rel="noopener">MySQL 大量sleeping before entering InnoDB 故障诊断</a> （很详细的一次问题排查步骤，里面有很多有用的命令）</li>
<li><a href="https://mp.weixin.qq.com/s/Edn_gPwcAHo5sYIzLJghzA" target="_blank" rel="noopener">哪些因素会导致慢查询？</a> （<font color="red">从应用到数据库，非常全面的列出了变慢的原因</font> 加一个：limit 时匹配不到数据会一直扫描。）</li>
<li><a href="https://mp.weixin.qq.com/s/AC9CwpSUTW6fXA-5mHxPGQ" target="_blank" rel="noopener">由MySQL复制延迟说起</a> （分析了复制延迟的主要原因和解决办法）</li>
<li><a href="https://mp.weixin.qq.com/s/safm3DQh2zYO9Pdk9ZXbkA" target="_blank" rel="noopener">为何COUNT很慢却不写SLOW LOG</a>：slow log记录规则</li>
<li><a href="https://mp.weixin.qq.com/s/7L-aBKmjORJ7Mg72Od47lw" target="_blank" rel="noopener">故障分析 | MySQL 优化案例 - select count(*)</a></li>
</ul>
<p>SQL变慢的原因主要有：</p>
<ol>
<li>SQL执行过程走索引不合理，导致执行缓慢。</li>
<li>使用合理索引，但是获取数据量比较多。(排序，临时表等)</li>
<li>网络重传丢包导致SQL变慢。</li>
<li>并发比较高的场景，请求排队处理，等待时间长。</li>
</ol>
<p>性能优化是一个老生常谈的问题，需要对相关流程和机制有很深入的研究才能对症下药。平时扩充、积累相关知识，在遇到问题后，才会有思路。经验很重要，但更多的是建立在平时的积累上。</p>
<h2 id="死锁系列"><a href="#死锁系列" class="headerlink" title="死锁系列"></a>死锁系列</h2><ul>
<li><a href="https://mp.weixin.qq.com/s/xUgzKPMZP777jdytrCL7oA" target="_blank" rel="noopener">漫谈死锁</a> （1. RC级别下也会存在Next-key 2. RC级别下，获取到不符合记录时会释放锁）<ul>
<li>innodb_print_all_deadlocks (将死锁信息记录到errorlog中参数)</li>
<li>innodb_status_output_locks (标准监控开关参数)</li>
</ul>
</li>
<li><a href="https://mp.weixin.qq.com/s/8XXimj2AIsRWj0fF-_iz3g" target="_blank" rel="noopener">如何阅读死锁日志</a> （锁组合）</li>
<li><a href="https://mp.weixin.qq.com/s/2GIrpSKHe3Y3KWMfDbA7jw" target="_blank" rel="noopener">MySQL 各种SQL语句加锁分析</a><ul>
<li>innodb_locks_unsafe_for_binlog</li>
</ul>
</li>
<li><a href="https://mp.weixin.qq.com/s/MywiDdffz13i8auSv4Xs9w" target="_blank" rel="noopener">死锁案例一</a> （简单插入唯一索引插入意向锁案例）</li>
<li><a href="https://mp.weixin.qq.com/s/7DehY-zQV1XHYufaII4oJg" target="_blank" rel="noopener">死锁案例二</a> （delete操作会有锁区间行为。PS：删除已有记录会导致锁范围变大)</li>
<li><a href="https://mp.weixin.qq.com/s/R1dJY4leZh5qRpUkvOsNBw" target="_blank" rel="noopener">死锁案例三</a> （insert同一区间时，唯一索引的插入意向锁导致的死锁）</li>
<li><a href="https://mp.weixin.qq.com/s/wXH9XFOK8wui7dDtR6Hzww" target="_blank" rel="noopener">死锁案例四</a> （并发插入时，插入意向锁不同阶段死锁。<font color="red">注意跟案例三的区别：本案例为X锁等待，案例三为S锁等待</font>)</li>
<li><a href="https://mp.weixin.qq.com/s/XCdLfyjSuWAEcWIjbcVfvg" target="_blank" rel="noopener">死锁案例五</a> （repalce into插入死锁情况）</li>
<li><a href="https://mp.weixin.qq.com/s/Lp7b2TGyFY-4uEN_M6Y8zg" target="_blank" rel="noopener">死锁案例六</a> （不同事务加锁顺序不一样，导致锁资源交叉影响导致的死锁)</li>
<li><a href="https://mp.weixin.qq.com/s/pC4XCpUOuNPMFj6UjuhWvw" target="_blank" rel="noopener">死锁案例七</a> （RC级别下，update不存在记录会持有lock_S + GAP锁）</li>
<li><a href="https://mp.weixin.qq.com/s/96CDhpgu5uUQ7qKYhKgt3w" target="_blank" rel="noopener">死锁案例八</a> （多并发插入下的又一插入意向锁竞争导致的死锁案例）</li>
<li><a href="https://mp.weixin.qq.com/s/h4Mwrowft2KKdTkqmRlYqw" target="_blank" rel="noopener">死锁案例九</a> （并发批量插入，导致的死锁）</li>
<li><a href="https://mp.weixin.qq.com/s/b9gNbdEHV3NNQrV9PKDPSw" target="_blank" rel="noopener">死锁案例十</a> （并发更新普通二级索引造成的死锁。PS：<font color="red">另一程度上说明索引不是随便加就行</font>）</li>
<li><a href="https://mp.weixin.qq.com/s/vMGlRtMQckJ1LUZaSBn5cw" target="_blank" rel="noopener">死锁案例十一</a> （<font color="red">5.6升级到5.7后，INSERT INTO .. ON DUPLICATE KEY语句锁模式加强</font>）</li>
<li><a href="https://mp.weixin.qq.com/s/SFdPDPhI270tnVI3uuOH0Q" target="_blank" rel="noopener">死锁案例十二</a> （唯一索引，多插入,插入意向锁死锁情况）</li>
<li><a href="!https://www.cnblogs.com/yulibostu/articles/9844061.html">一个最不可思议的死锁分析</a> （并发delete语句造成的死锁情况，里面引用了很多好文章，需要细品）</li>
</ul>
<p>分析死锁单从<code>show engines innodb status</code>中往往只能看到最后执行的语句，并不能很好的了解整个死锁形成过程。所以在分析死锁时，需要先了解两个死锁事务的具体逻辑和语句过程。<br>了解MySQL不同操作的加锁过程非常重要，这是对死锁形成过程分析的基础。</p>
</summary>
<category term="MyQL" scheme="http://yoursite.com/categories/MyQL/"/>
<category term="MySQL" scheme="http://yoursite.com/tags/MySQL/"/>
</entry>
<entry>
<title>Redis 6.0新特性-客户端缓存</title>
<link href="http://yoursite.com/2020/07/29/new/Redis/Redis-6-0%E6%96%B0%E7%89%B9%E6%80%A7-%E5%AE%A2%E6%88%B7%E7%AB%AF%E7%BC%93%E5%AD%98/"/>
<id>http://yoursite.com/2020/07/29/new/Redis/Redis-6-0新特性-客户端缓存/</id>
<published>2020-07-29T11:52:38.000Z</published>
<updated>2020-07-29T11:53:27.676Z</updated>
<content type="html"><![CDATA[<blockquote><p>你见过5点半的曼哈顿街头吗？一年前，当antirez在参加完纽约Redis大会后，5:30就在旅店中醒来了，面对曼哈顿街头的美丽景色，在芸芸众生中思索Redis的未来。包括客户端缓存。</p></blockquote><h2 id="简介"><a href="#简介" class="headerlink" title="简介"></a>简介</h2><p>Redis实现的是一个服务端协助的客户端缓存，叫做tracking。用法是：<br><code>CLIENT TRACKING ON|OFF [REDIRECT client-id] [PREFIX prefix] [BCAST] [OPTIN] [OPTOUT] [NOLOOP]</code><br>当tracking开启时，Redis客户端会“记住”每个请求过的key；当服务端key发生修改命令操作时，就会发送失效消息给客户端。失效消息可以通过RESP3协议发送给请求的客户端，或者转发给一个订阅频道<code>__redis__:invalidate</code>的连接。</p><ul><li>REDIRECT：将失效消息转发给另外一个客户端。当我们使用的是老的RESP2和Redis通讯时，client本身不支持处理失效消息，所以可以开启一个Pub/Sub客户端处理失效消息。</li><li>BCAST：使用广播模式开始tracking，在这种模式下客户端需要设置将track的key前缀，这些key的失效消息会广播给所有参与的客户端，不管这些客户端是否请求/缓存了这些key。不开启广播模式的时候，Redis只会Track那些只读命令请求的key，并且只会报告一次失效消息。</li><li>PREFIX：只应用了广播模式，注册一个key的前缀。所有以这个前缀开始的key有修改时，都会发送失效消息。可以注册多个前缀，如果不设置前缀，那么广播模式会track每一个key。</li><li>OPTIN：当广播模式没有开启时，正常不会track只读命令的key，除非它们在CLIENT CACHING yes之后被调用。</li><li>OPTPUT：当广播模式没有开启时，正常会track只读命令的key，除非它们在CLIENT CACHING off之后被调用。</li><li>NOLOOP：不发送自身修改的key失效消息给自己。</li></ul><a id="more"></a><h2 id="使用验证"><a href="#使用验证" class="headerlink" title="使用验证"></a>使用验证</h2><p>为了能够更好的观察，我们使用telnet来进行连接。并开启RESP3协议模式<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">telnet 127.0.0.1 6379</span><br><span class="line">hello 3</span><br></pre></td></tr></table></figure></p><h3 id="BCAST-广播模式"><a href="#BCAST-广播模式" class="headerlink" title="BCAST 广播模式"></a>BCAST 广播模式</h3><p>在使用telnet连接redis服务后，使用命令<code>client tracking on</code>启动广播模式进行。并使用<code>get a</code>命令模拟读请求。<br>在另外一个客户端对该key进行修改后，telnet这个client立即会收到该key过期的消息通知。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line">hello 3</span><br><span class="line">%7</span><br><span class="line">$6</span><br><span class="line">server</span><br><span class="line">$5</span><br><span class="line">redis</span><br><span class="line">$7</span><br><span class="line">version</span><br><span class="line">$5</span><br><span class="line">6.0.5</span><br><span class="line">$5</span><br><span class="line">proto</span><br><span class="line">:3</span><br><span class="line">...</span><br><span class="line"></span><br><span class="line">client tracking on</span><br><span class="line">+OK</span><br><span class="line">get a</span><br><span class="line">$1</span><br><span class="line">2</span><br><span class="line">>2</span><br><span class="line">$10</span><br><span class="line">invalidate</span><br><span class="line">*1</span><br><span class="line">$1</span><br><span class="line">a</span><br></pre></td></tr></table></figure></p><p>在收到key的实效消息通知后，如果其他客户端再对该key进行修改。telnet也不会再收到实效消息。这意味telnet客户端已经将该key的缓存给清理了，除非再次进行读请求将该key再次进行缓存。</p><p>在telnet运行中，随时可以使用命令<code>client tracking off</code>关闭广播模式，telnet客户端将不会再缓存key。</p><h3 id="PREFIX-前缀模式"><a href="#PREFIX-前缀模式" class="headerlink" title="PREFIX 前缀模式"></a>PREFIX 前缀模式</h3><p>在上面的广播模式下，客户端会缓存所有的请求过的key。如果你只想跟踪特定的key，则可以使用前缀匹配的方式。<br>使用以下命令只对指定前缀的key进行缓存，可以执行多次，指定不同前缀。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">client tracking on prefix a bcast</span><br><span class="line">client tracking on prefix user bcast</span><br></pre></td></tr></table></figure></p><p>你可以使用<code>client tracking off</code>取消对所有key前缀的缓存。但是不支持只停止特定单个前缀key的缓存。</p><h3 id="OPTIN-选择加入"><a href="#OPTIN-选择加入" class="headerlink" title="OPTIN 选择加入"></a>OPTIN 选择加入</h3><p>使用<code>optin</code>可以有选择的开启tracking。只有你发送<code>client caching yes</code>之后的下一条只读命令涉及的key才会被tracking。其他只读命令中的key不会被tracking。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">client tracking on optin</span><br><span class="line">+OK</span><br><span class="line"></span><br><span class="line">get a</span><br><span class="line">$1</span><br><span class="line">2</span><br><span class="line"></span><br><span class="line">client caching yes</span><br><span class="line">+OK</span><br><span class="line">get a</span><br><span class="line">$4</span><br><span class="line">1000</span><br><span class="line"></span><br><span class="line">>2</span><br><span class="line">$10</span><br><span class="line">invalidate</span><br><span class="line">*1</span><br><span class="line">$1</span><br><span class="line">a</span><br></pre></td></tr></table></figure></p><h3 id="OPTOUT-选择退出"><a href="#OPTOUT-选择退出" class="headerlink" title="OPTOUT 选择退出"></a>OPTOUT 选择退出</h3><p>使用<code>optout</code>可以有选择的关闭tracking。当你发送<code>client caching off</code>之后的下一条只读命令涉及的key不会被tracking。其他只读命令中的key都会被tracking。它的效果与<code>optin</code>相反。</p><h3 id="REDIRECT"><a href="#REDIRECT" class="headerlink" title="REDIRECT"></a>REDIRECT</h3><p>REDIRECT是为了兼容RESP2协议的一个处理方式，能够将失效消息转发给另外一个client。</p><p>RESP2协议客户端，查看客户端ID，并开启订阅模式<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">client id</span><br><span class="line">:135</span><br><span class="line">SUBSCRIBE __redis__:invalidate</span><br><span class="line">*3</span><br><span class="line">$9</span><br><span class="line">subscribe</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">---</span><br><span class="line">$20</span><br><span class="line">__redis__:invalidate</span><br><span class="line">:1</span><br><span class="line">*3</span><br><span class="line">$7</span><br><span class="line">message</span><br><span class="line">$20</span><br><span class="line">__redis__:invalidate</span><br><span class="line">*1</span><br><span class="line">$1</span><br><span class="line">a</span><br></pre></td></tr></table></figure></p><p>RESP3开启失效消息转发，并执行修改操作。在执行修改操作后，RESP2客户端就收到了失效消息。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">client tracking on bcast redirect 135</span><br><span class="line">+OK</span><br><span class="line">set a 300</span><br><span class="line">+OK</span><br></pre></td></tr></table></figure></p><p>如果转发的目的client开启了RESP3协议，就不需要RESP3 Pub/Sub了，因为RESP3原生支持Push消息。</p><h3 id="NOLOOP"><a href="#NOLOOP" class="headerlink" title="NOLOOP"></a>NOLOOP</h3><p>正常设置时，失效消息是发给所有参与的Client，但是如果设置了NOLOOP，则不会发送给更新这个key的Client。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line">client tracking on bcast</span><br><span class="line">+OK</span><br><span class="line">get a</span><br><span class="line">$1</span><br><span class="line">2</span><br><span class="line">set a 1</span><br><span class="line">+OK</span><br><span class="line">>2</span><br><span class="line">$10</span><br><span class="line">invalidate</span><br><span class="line">*1</span><br><span class="line">$1</span><br><span class="line">a</span><br><span class="line"></span><br><span class="line">----</span><br><span class="line">client tracking on bcast noloop</span><br><span class="line">+OK</span><br><span class="line">set a 2</span><br><span class="line">+OK</span><br><span class="line">get a</span><br><span class="line">$1</span><br><span class="line">2</span><br><span class="line">set a 1</span><br><span class="line">+OK</span><br></pre></td></tr></table></figure></p>]]></content>
<summary type="html">
<blockquote>
<p>你见过5点半的曼哈顿街头吗？一年前，当antirez在参加完纽约Redis大会后，5:30就在旅店中醒来了，面对曼哈顿街头的美丽景色，在芸芸众生中思索Redis的未来。包括客户端缓存。</p>
</blockquote>
<h2 id="简介"><a href="#简介" class="headerlink" title="简介"></a>简介</h2><p>Redis实现的是一个服务端协助的客户端缓存，叫做tracking。用法是：<br><code>CLIENT TRACKING ON|OFF [REDIRECT client-id] [PREFIX prefix] [BCAST] [OPTIN] [OPTOUT] [NOLOOP]</code><br>当tracking开启时，Redis客户端会“记住”每个请求过的key；当服务端key发生修改命令操作时，就会发送失效消息给客户端。失效消息可以通过RESP3协议发送给请求的客户端，或者转发给一个订阅频道<code>__redis__:invalidate</code>的连接。</p>
<ul>
<li>REDIRECT：将失效消息转发给另外一个客户端。当我们使用的是老的RESP2和Redis通讯时，client本身不支持处理失效消息，所以可以开启一个Pub/Sub客户端处理失效消息。</li>
<li>BCAST：使用广播模式开始tracking，在这种模式下客户端需要设置将track的key前缀，这些key的失效消息会广播给所有参与的客户端，不管这些客户端是否请求/缓存了这些key。不开启广播模式的时候，Redis只会Track那些只读命令请求的key，并且只会报告一次失效消息。</li>
<li>PREFIX：只应用了广播模式，注册一个key的前缀。所有以这个前缀开始的key有修改时，都会发送失效消息。可以注册多个前缀，如果不设置前缀，那么广播模式会track每一个key。</li>
<li>OPTIN：当广播模式没有开启时，正常不会track只读命令的key，除非它们在CLIENT CACHING yes之后被调用。</li>
<li>OPTPUT：当广播模式没有开启时，正常会track只读命令的key，除非它们在CLIENT CACHING off之后被调用。</li>
<li>NOLOOP：不发送自身修改的key失效消息给自己。</li>
</ul>
</summary>
<category term="Redis" scheme="http://yoursite.com/categories/Redis/"/>
<category term="Redis" scheme="http://yoursite.com/tags/Redis/"/>
<category term="Redis6.0" scheme="http://yoursite.com/tags/Redis6-0/"/>
</entry>
<entry>
<title>Redis ACL从使用到实现</title>
<link href="http://yoursite.com/2020/07/20/new/Redis/Redis-ACL%E4%BB%8E%E4%BD%BF%E7%94%A8%E5%88%B0%E5%AE%9E%E7%8E%B0/"/>
<id>http://yoursite.com/2020/07/20/new/Redis/Redis-ACL从使用到实现/</id>
<published>2020-07-20T12:02:31.000Z</published>
<updated>2020-07-29T11:55:04.544Z</updated>
<content type="html"><![CDATA[<h2 id="简介"><a href="#简介" class="headerlink" title="简介"></a>简介</h2><p>Redis ACL是6.0版本中推出的新功能。是一项可以限制用户连接可执行命令和键访问操作的功能。<br>客户端在连接服务器之后，客户端需要提供用户名和密码用于验证身份，如果验证成功，客户端连接会关联特定用户以及用户相应的权限。Redis可以配置新的客户端连接自动使用默认用户(default)进行验证(默认选项)。<br>此外，ACL功能对旧版客户端和应用都是向后兼容的，对于旧版配置用户密码的方式(requirepass)也是支持的。该命令设置的是default用户的密码，即<code>auth password</code> 等价于 <code>auth default password</code>。没有指定用户的客户端连接使用的都是default，可以通过控制defalut用户权限来兼容老版本。</p><h2 id="ACL使用场景"><a href="#ACL使用场景" class="headerlink" title="ACL使用场景"></a>ACL使用场景</h2><ol><li>你希望限制用户访问命令和键以提高安全性。不在信任列表中的用户没有权限访问，而在信任列表中的用户拥有完成工作的最小访问权限。例如一些客户端只可以执行只读的命令。</li><li>你希望提供运维安全。避免程序出错或者人为操作失误导致数据或者配置受到损坏。比如执行flush、keys等。在之前版本中都是通过命令重命令的方式来对使用者隐藏，但是这样可能会存在其他隐患。</li></ol><h2 id="ACL使用"><a href="#ACL使用" class="headerlink" title="ACL使用"></a>ACL使用</h2><h3 id="ACL规则"><a href="#ACL规则" class="headerlink" title="ACL规则"></a>ACL规则</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">on：启用用户</span><br><span class="line">off：禁止用户</span><br><span class="line">+：添加命令到用户允许执行命令列表</span><br><span class="line">-：从用户允许执行命令列表删除命令</span><br><span class="line">+@：允许用户执行定义在category类别中的命令</span><br><span class="line">-@：删除用户拥有的category类别命令权限</span><br><span class="line">+|subcommand：允许用户启用一个被禁止类别下的子命令。</span><br><span class="line">allcommands：+@all 的别名. </span><br><span class="line">nocommands：-@all 的别名。</span><br><span class="line">~：添加一个键值模式的权限</span><br><span class="line">allkeys：是 ~* 的别名</span><br><span class="line">resetkeys：在键模式列表里面清空所有的键模式</span><br><span class="line">>：添加密码到用户有效密码列表里</span><br><span class="line"><：从用户有效密码列表中删除密码</span><br><span class="line">#：添加 SHA-256 形式哈希值到用户有效密码列表里</span><br><span class="line">!：从有效的密码列表中删除哈希值密码</span><br><span class="line">nopass：删除所有与用户关联的密码</span><br><span class="line">resetpass：清除用户所有密码</span><br><span class="line">reset：用户将返回和它被默认创建时同样的状态。</span><br></pre></td></tr></table></figure><h3 id="ACL命令使用"><a href="#ACL命令使用" class="headerlink" title="ACL命令使用"></a>ACL命令使用</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">/* ACL -- show and modify the configuration of ACL users.</span><br><span class="line"> * ACL HELP // 获取帮助信息</span><br><span class="line"> * ACL LOAD // 从外部ACL文件导入用户信息</span><br><span class="line"> * ACL LIST // 所有用户权限信息</span><br><span class="line"> * ACL USERS // 所有用户的列表</span><br><span class="line"> * ACL CAT [<category>] // 命令分类，及分类下的具体命令列表</span><br><span class="line"> * ACL SETUSER <username> ... acl rules ... // 添加用户、权限修改等操作</span><br><span class="line"> * ACL DELUSER <username> [...] // 删除用户</span><br><span class="line"> * ACL GETUSER <username> // 获取指定用户的信息</span><br><span class="line"> * ACL GENPASS // 生成一个密码</span><br><span class="line"> * ACL WHOAMI // 获取当前用户</span><br><span class="line"> */</span><br></pre></td></tr></table></figure><a id="more"></a><h3 id="使用示例"><a href="#使用示例" class="headerlink" title="使用示例"></a>使用示例</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">// 创建一个未启用的用户</span><br><span class="line">127.0.0.1:6379> acl setuser demo</span><br><span class="line">OK</span><br><span class="line"></span><br><span class="line">// 查看所有用户</span><br><span class="line">127.0.0.1:6379> acl list</span><br><span class="line">1) "user alice on >1234 ~alice:* +@all"</span><br><span class="line">2) "user default on nopass ~* +@all"</span><br><span class="line">3) "user demo off -@all"</span><br><span class="line"></span><br><span class="line">// 启用用户，设置密码为1234，并授予demo:* 键值对的所有权限</span><br><span class="line">127.0.0.1:6379> acl setuser demo on >1234 ~demo:* +@all</span><br><span class="line">OK</span><br><span class="line">127.0.0.1:6379> acl list</span><br><span class="line">1) "user alice on >1234 ~alice:* +@all"</span><br><span class="line">2) "user default on nopass ~* +@all"</span><br><span class="line">3) "user demo on >1234 ~demo:* +@all"</span><br></pre></td></tr></table></figure><h2 id="实现"><a href="#实现" class="headerlink" title="实现"></a>实现</h2><p>Redis新增了一个user结构体。结构体具体定义如下。<br>需要注意的是：</p><ol><li>Redis使用allowed_commands数据来记录是否拥有命令的执行权限。每一种命令对应其中的一个字节(使用全局的一棵rax树保存命令，id获取函数为<code>ACLGetCommandID</code>)。</li><li>密码使用的是一个list列表。表示一个用户允许多个密码登录。<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line">// 用户定义</span><br><span class="line">typedef struct user {</span><br><span class="line"> // 用户名</span><br><span class="line"> sds name; /* The username as an SDS string. */</span><br><span class="line"> // 用户标记</span><br><span class="line"> uint64_t flags; /* See USER_FLAG_* */</span><br><span class="line"></span><br><span class="line"> /* The bit in allowed_commands is set if this user has the right to</span><br><span class="line"> * execute this command. In commands having subcommands, if this bit is</span><br><span class="line"> * set, then all the subcommands are also available.</span><br><span class="line"> *</span><br><span class="line"> * If the bit for a given command is NOT set and the command has</span><br><span class="line"> * subcommands, Redis will also check allowed_subcommands in order to</span><br><span class="line"> * understand if the command can be executed. */</span><br><span class="line"> // 允许执行的命令数组</span><br><span class="line"> uint64_t allowed_commands[USER_COMMAND_BITS_COUNT/64];</span><br><span class="line"></span><br><span class="line"> /* This array points, for each command ID (corresponding to the command</span><br><span class="line"> * bit set in allowed_commands), to an array of SDS strings, terminated by</span><br><span class="line"> * a NULL pointer, with all the sub commands that can be executed for</span><br><span class="line"> * this command. When no subcommands matching is used, the field is just</span><br><span class="line"> * set to NULL to avoid allocating USER_COMMAND_BITS_COUNT pointers. */</span><br><span class="line"> // 一个数组指针，表示在命令ID下允许执行的具体命令列表，以NULL结束。</span><br><span class="line"> // 如果没有允许执行的命令，这个值为NULL，避免分配USER_COMMAND_BITS_COUNT空间。</span><br><span class="line"> sds **allowed_subcommands;</span><br><span class="line"> // 密码</span><br><span class="line"> list *passwords; /* A list of SDS valid passwords for this user. */</span><br><span class="line"> // 运行执行命令的匹配符列表。如果这个值为NULL表示不允许执行任何命令。除非这个用户的flag为ALLKEYS。</span><br><span class="line"> list *patterns; /* A list of allowed key patterns. If this field is NULL</span><br><span class="line"> the user cannot mention any key in a command, unless</span><br><span class="line"> the flag ALLKEYS is set in the user. */</span><br><span class="line">} user;</span><br></pre></td></tr></table></figure></li></ol><p>acl命令具体实现在<code>acl.c -> aclCommand()</code>，下面我们将通过这个函数来分析acl命令的具体实现。</p><h3 id="增加用户及权限"><a href="#增加用户及权限" class="headerlink" title="增加用户及权限"></a>增加用户及权限</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line">if (!strcasecmp(sub,"setuser") && c->argc >= 3) {</span><br><span class="line"> sds username = c->argv[2]->ptr;</span><br><span class="line"> /* Create a temporary user to validate and stage all changes against</span><br><span class="line"> * before applying to an existing user or creating a new user. If all</span><br><span class="line"> * arguments are valid the user parameters will all be applied together.</span><br><span class="line"> * If there are any errors then none of the changes will be applied. */</span><br><span class="line"> // 创建一个临时的用户去验证执行阶段，如果在某个阶段存在问题也不会影响原用户。</span><br><span class="line"> user *tempu = ACLCreateUnlinkedUser();</span><br><span class="line"> </span><br><span class="line"> // 判断该用户是否已存在，如果存在。将权限同步给临时用户</span><br><span class="line"> user *u = ACLGetUserByName(username,sdslen(username));</span><br><span class="line"> if (u) ACLCopyUser(tempu, u);</span><br><span class="line"></span><br><span class="line"> for (int j = 3; j < c->argc; j++) {</span><br><span class="line"> // 根据传参执行授权操作</span><br><span class="line"> if (ACLSetUser(tempu,c->argv[j]->ptr,sdslen(c->argv[j]->ptr)) != C_OK) {</span><br><span class="line"> char *errmsg = ACLSetUserStringError();</span><br><span class="line"> addReplyErrorFormat(c,</span><br><span class="line"> "Error in ACL SETUSER modifier '%s': %s",</span><br><span class="line"> (char*)c->argv[j]->ptr, errmsg);</span><br><span class="line"></span><br><span class="line"> ACLFreeUser(tempu);</span><br><span class="line"> return;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> /* Overwrite the user with the temporary user we modified above. */</span><br><span class="line"> if (!u) u = ACLCreateUser(username,sdslen(username));</span><br><span class="line"> serverAssert(u != NULL);</span><br><span class="line"> ACLCopyUser(u, tempu);</span><br><span class="line"> ACLFreeUser(tempu);</span><br><span class="line"> addReply(c,shared.ok);</span><br></pre></td></tr></table></figure><p><code>acl setuser</code>是Redis用来增加用户及权限的命令。我们可以看到在授权中，会先创建一个临时用户，先在临时用户上进行权限修改操作。如果在某阶段执行出错，前面执行成功的权限修改行为并不会影响到原用户。<br>调用的关键函数为：</p><ul><li><code>ACLSetUser</code>：按照ACL规则来进行具体的授权行为。</li><li><code>ACLCopyUser</code>：将一个用户的所有权限参数拷贝给一个新用户。</li></ul><p>其余功能实现就不介绍了。。</p><h3 id="权限检查"><a href="#权限检查" class="headerlink" title="权限检查"></a>权限检查</h3><p>在所有命令执行前，都会调用<code>processCommand</code>函数。在这个函数中，会进行一些命令执行前的常规检查，比如参数个数、实例状态、用户权限等。<br>该函数会调用<code>acl.c -> ACLCheckCommandPerm()</code>函数来检查用户是否拥有命令和键值对的执行权限。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br></pre></td><td class="code"><pre><span class="line">/* Check if the command ready to be excuted in the client 'c', and already</span><br><span class="line"> * referenced by c->cmd, can be executed by this client according to the</span><br><span class="line"> * ACls associated to the client user c->user.</span><br><span class="line"> *</span><br><span class="line"> * If the user can execute the command ACL_OK is returned, otherwise</span><br><span class="line"> * ACL_DENIED_CMD or ACL_DENIED_KEY is returned: the first in case the</span><br><span class="line"> * command cannot be executed because the user is not allowed to run such</span><br><span class="line"> * command, the second if the command is denied because the user is trying</span><br><span class="line"> * to access keys that are not among the specified patterns. */</span><br><span class="line">// 检查当前客户端是否拥有执行命令的权限</span><br><span class="line">int ACLCheckCommandPerm(client *c) {</span><br><span class="line"> user *u = c->user;</span><br><span class="line"> uint64_t id = c->cmd->id;</span><br><span class="line"></span><br><span class="line"> /* If there is no associated user, the connection can run anything. */</span><br><span class="line"> if (u == NULL) return ACL_OK;</span><br><span class="line"></span><br><span class="line"> /* Check if the user can execute this command. */</span><br><span class="line"> // 检查是否拥有执行命令的权限</span><br><span class="line"> if (!(u->flags & USER_FLAG_ALLCOMMANDS) &&</span><br><span class="line"> c->cmd->proc != authCommand)</span><br><span class="line"> {</span><br><span class="line"> /* If the bit is not set we have to check further, in case the</span><br><span class="line"> * command is allowed just with that specific subcommand. */</span><br><span class="line"> if (ACLGetUserCommandBit(u,id) == 0) {</span><br><span class="line"> /* Check if the subcommand matches. */</span><br><span class="line"> if (c->argc < 2 ||</span><br><span class="line"> u->allowed_subcommands == NULL ||</span><br><span class="line"> u->allowed_subcommands[id] == NULL)</span><br><span class="line"> {</span><br><span class="line"> return ACL_DENIED_CMD;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> long subid = 0;</span><br><span class="line"> while (1) {</span><br><span class="line"> if (u->allowed_subcommands[id][subid] == NULL)</span><br><span class="line"> return ACL_DENIED_CMD;</span><br><span class="line"> if (!strcasecmp(c->argv[1]->ptr,</span><br><span class="line"> u->allowed_subcommands[id][subid]))</span><br><span class="line"> break; /* Subcommand match found. Stop here. */</span><br><span class="line"> subid++;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> /* Check if the user can execute commands explicitly touching the keys</span><br><span class="line"> * mentioned in the command arguments. */</span><br><span class="line"> // 检查是否拥有键值对操作的权限</span><br><span class="line"> if (!(c->user->flags & USER_FLAG_ALLKEYS) &&</span><br><span class="line"> (c->cmd->getkeys_proc || c->cmd->firstkey))</span><br><span class="line"> {</span><br><span class="line"> int numkeys;</span><br><span class="line"> int *keyidx = getKeysFromCommand(c->cmd,c->argv,c->argc,&numkeys);</span><br><span class="line"> for (int j = 0; j < numkeys; j++) {</span><br><span class="line"> listIter li;</span><br><span class="line"> listNode *ln;</span><br><span class="line"> listRewind(u->patterns,&li);</span><br><span class="line"></span><br><span class="line"> /* Test this key against every pattern. */</span><br><span class="line"> int match = 0;</span><br><span class="line"> while((ln = listNext(&li))) {</span><br><span class="line"> sds pattern = listNodeValue(ln);</span><br><span class="line"> size_t plen = sdslen(pattern);</span><br><span class="line"> int idx = keyidx[j];</span><br><span class="line"> if (stringmatchlen(pattern,plen,c->argv[idx]->ptr,</span><br><span class="line"> sdslen(c->argv[idx]->ptr),0))</span><br><span class="line"> {</span><br><span class="line"> match = 1;</span><br><span class="line"> break;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> if (!match) {</span><br><span class="line"> getKeysFreeResult(keyidx);</span><br><span class="line"> return ACL_DENIED_KEY;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> getKeysFreeResult(keyidx);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> /* If we survived all the above checks, the user can execute the</span><br><span class="line"> * command. */</span><br><span class="line"> return ACL_OK;</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><blockquote><p><a href="https://www.infoq.cn/article/fe97cBWx6Hqk9qVvoW7r" target="_blank" rel="noopener">https://www.infoq.cn/article/fe97cBWx6Hqk9qVvoW7r</a></p></blockquote>]]></content>
<summary type="html">
<h2 id="简介"><a href="#简介" class="headerlink" title="简介"></a>简介</h2><p>Redis ACL是6.0版本中推出的新功能。是一项可以限制用户连接可执行命令和键访问操作的功能。<br>客户端在连接服务器之后，客户端需要提供用户名和密码用于验证身份，如果验证成功，客户端连接会关联特定用户以及用户相应的权限。Redis可以配置新的客户端连接自动使用默认用户(default)进行验证(默认选项)。<br>此外，ACL功能对旧版客户端和应用都是向后兼容的，对于旧版配置用户密码的方式(requirepass)也是支持的。该命令设置的是default用户的密码，即<code>auth password</code> 等价于 <code>auth default password</code>。没有指定用户的客户端连接使用的都是default，可以通过控制defalut用户权限来兼容老版本。</p>
<h2 id="ACL使用场景"><a href="#ACL使用场景" class="headerlink" title="ACL使用场景"></a>ACL使用场景</h2><ol>
<li>你希望限制用户访问命令和键以提高安全性。不在信任列表中的用户没有权限访问，而在信任列表中的用户拥有完成工作的最小访问权限。例如一些客户端只可以执行只读的命令。</li>
<li>你希望提供运维安全。避免程序出错或者人为操作失误导致数据或者配置受到损坏。比如执行flush、keys等。在之前版本中都是通过命令重命令的方式来对使用者隐藏，但是这样可能会存在其他隐患。</li>
</ol>
<h2 id="ACL使用"><a href="#ACL使用" class="headerlink" title="ACL使用"></a>ACL使用</h2><h3 id="ACL规则"><a href="#ACL规则" class="headerlink" title="ACL规则"></a>ACL规则</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">on：启用用户</span><br><span class="line">off：禁止用户</span><br><span class="line">+：添加命令到用户允许执行命令列表</span><br><span class="line">-：从用户允许执行命令列表删除命令</span><br><span class="line">+@：允许用户执行定义在category类别中的命令</span><br><span class="line">-@：删除用户拥有的category类别命令权限</span><br><span class="line">+|subcommand：允许用户启用一个被禁止类别下的子命令。</span><br><span class="line">allcommands：+@all 的别名. </span><br><span class="line">nocommands：-@all 的别名。</span><br><span class="line">~：添加一个键值模式的权限</span><br><span class="line">allkeys：是 ~* 的别名</span><br><span class="line">resetkeys：在键模式列表里面清空所有的键模式</span><br><span class="line">&gt;：添加密码到用户有效密码列表里</span><br><span class="line">&lt;：从用户有效密码列表中删除密码</span><br><span class="line">#：添加 SHA-256 形式哈希值到用户有效密码列表里</span><br><span class="line">!：从有效的密码列表中删除哈希值密码</span><br><span class="line">nopass：删除所有与用户关联的密码</span><br><span class="line">resetpass：清除用户所有密码</span><br><span class="line">reset：用户将返回和它被默认创建时同样的状态。</span><br></pre></td></tr></table></figure>
<h3 id="ACL命令使用"><a href="#ACL命令使用" class="headerlink" title="ACL命令使用"></a>ACL命令使用</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">/* ACL -- show and modify the configuration of ACL users.</span><br><span class="line"> * ACL HELP // 获取帮助信息</span><br><span class="line"> * ACL LOAD // 从外部ACL文件导入用户信息</span><br><span class="line"> * ACL LIST // 所有用户权限信息</span><br><span class="line"> * ACL USERS // 所有用户的列表</span><br><span class="line"> * ACL CAT [&lt;category&gt;] // 命令分类，及分类下的具体命令列表</span><br><span class="line"> * ACL SETUSER &lt;username&gt; ... acl rules ... // 添加用户、权限修改等操作</span><br><span class="line"> * ACL DELUSER &lt;username&gt; [...] // 删除用户</span><br><span class="line"> * ACL GETUSER &lt;username&gt; // 获取指定用户的信息</span><br><span class="line"> * ACL GENPASS // 生成一个密码</span><br><span class="line"> * ACL WHOAMI // 获取当前用户</span><br><span class="line"> */</span><br></pre></td></tr></table></figure>
</summary>
<category term="Redis" scheme="http://yoursite.com/categories/Redis/"/>
<category term="Redis" scheme="http://yoursite.com/tags/Redis/"/>
<category term="Redis6.0" scheme="http://yoursite.com/tags/Redis6-0/"/>
</entry>
<entry>
<title>从Redis源码 acl.c->time_independent_strcmp() 实现谈到 定时攻击</title>
<link href="http://yoursite.com/2020/07/14/%E5%AE%9A%E6%97%B6%E6%94%BB%E5%87%BB/"/>
<id>http://yoursite.com/2020/07/14/定时攻击/</id>
<published>2020-07-14T05:40:13.000Z</published>
<updated>2020-07-14T05:41:12.797Z</updated>
<content type="html"><![CDATA[<h2 id="问题引出"><a href="#问题引出" class="headerlink" title="问题引出"></a>问题引出</h2><p>在Redis阅读Redis源码的时候，发现acl.c中有个方法<code>time_independent_strcmp</code>。它的作用就是用来对比两个字符串是否相等，如果相等，那么就返回0，如果不相等，那么就返回非0值。<br>是不是很熟悉，没错！这跟C语言中的<code>strcmp</code>功能是一样的。那么为什么Redis的作者需要自己去实现一个呢？</p><p>我们来看一下这个函数的实现：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line">int time_independent_strcmp(char *a, char *b) {</span><br><span class="line"> char bufa[CONFIG_AUTHPASS_MAX_LEN], bufb[CONFIG_AUTHPASS_MAX_LEN];</span><br><span class="line"> /* The above two strlen perform len(a) + len(b) operations where either</span><br><span class="line"> * a or b are fixed (our password) length, and the difference is only</span><br><span class="line"> * relative to the length of the user provided string, so no information</span><br><span class="line"> * leak is possible in the following two lines of code. */</span><br><span class="line"> unsigned int alen = strlen(a);</span><br><span class="line"> unsigned int blen = strlen(b);</span><br><span class="line"> unsigned int j;</span><br><span class="line"> int diff = 0;</span><br><span class="line"></span><br><span class="line"> /* We can't compare strings longer than our static buffers.</span><br><span class="line"> * Note that this will never pass the first test in practical circumstances</span><br><span class="line"> * so there is no info leak. */</span><br><span class="line"> if (alen > sizeof(bufa) || blen > sizeof(bufb)) return 1;</span><br><span class="line"></span><br><span class="line"> memset(bufa,0,sizeof(bufa)); /* Constant time. */</span><br><span class="line"> memset(bufb,0,sizeof(bufb)); /* Constant time. */</span><br><span class="line"> /* Again the time of the following two copies is proportional to</span><br><span class="line"> * len(a) + len(b) so no info is leaked. */</span><br><span class="line"> memcpy(bufa,a,alen);</span><br><span class="line"> memcpy(bufb,b,blen);</span><br><span class="line"></span><br><span class="line"> /* Always compare all the chars in the two buffers without</span><br><span class="line"> * conditional expressions. */</span><br><span class="line"> for (j = 0; j < sizeof(bufa); j++) {</span><br><span class="line"> diff |= (bufa[j] ^ bufb[j]);</span><br><span class="line"> }</span><br><span class="line"> /* Length must be equal as well. */</span><br><span class="line"> diff |= alen ^ blen;</span><br><span class="line"> return diff; /* If zero strings are the same. */</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>可以看到，这个方法会先对比两个字符串的每一位，然后再判断两个字符串的长度是否相等。<br>看到这里，你是不是会想这样实现岂不是每次都会去对比所有字符，效率比<code>strcmp</code>还低！意义在哪呢？没错，我当时也是这么想的。于是去搜索了issue，于是有了意外的收获：<a href="https://github.com/redis/redis/issues/560" target="_blank" rel="noopener">https://github.com/redis/redis/issues/560</a>。<br>Redis作者原来实现也的确使用的是<code>strcmp</code>，在这个issue中，有人提出了《定时攻击》这个安全问题，于是Redis作者就这个问题进行了修复，也就是<code>time_independent_strcmp</code>方法的由来。</p><h2 id="定时攻击"><a href="#定时攻击" class="headerlink" title="定时攻击"></a>定时攻击</h2><p>简单来说，就是通过多次的密码攻击形式进行破解攻击，并根据目标服务返回的时间来确定当前字符是否正确。<br>在<code>strcmp</code>方法中，本质就是将两个字符串从头开始逐一比较，发现不同就立刻停止返回。假如输入的两个串开头有一部分相同，那么判断字符所耗费的时间就要变长一点。这好像就在告诉别人<font color="red">你输入的密码错误，但是前两个字符是正确的</font></p><p>解决这种攻击一般有几种方法：</p><ul><li>随机休眠时间</li><li>估算执行时间，固定返回时间</li><li>固定执行流程，返回时间不受具体字符影响。</li></ul><p>在Redis这种高性能服务中，随机休眠以及将命令执行时间固定这两种方式会严重的影响执行效率。<br>所以Redis的作者通过在密码比较时，无论正确与否都正常执行下去，尽可能的返回相同时间，以避免这种攻击来破解密码。</p><blockquote><p><a href="https://github.com/redis/redis/issues/560" target="_blank" rel="noopener">https://github.com/redis/redis/issues/560</a><br><a href="https://eyhn.in/%E5%AF%86%E7%A0%81%E5%AD%A6%E4%B8%AD%E6%81%92%E5%AE%9A%E6%97%B6%E9%97%B4%E8%AE%BE%E8%AE%A1%EF%BC%88%E8%AF%91%E6%96%87%EF%BC%89/" target="_blank" rel="noopener">密码学中恒定时间设计（译文）</a></p></blockquote>]]></content>
<summary type="html">
<h2 id="问题引出"><a href="#问题引出" class="headerlink" title="问题引出"></a>问题引出</h2><p>在Redis阅读Redis源码的时候，发现acl.c中有个方法<code>time_independent_strcmp</
</summary>
<category term="Redis" scheme="http://yoursite.com/categories/Redis/"/>
<category term="Redis" scheme="http://yoursite.com/tags/Redis/"/>
</entry>
<entry>
<title>Redis2.8 Bug记录一则</title>
<link href="http://yoursite.com/2020/07/08/new/Redis/Redis2-8-Bug%E8%AE%B0%E5%BD%95%E4%B8%80%E5%88%99/"/>
<id>http://yoursite.com/2020/07/08/new/Redis/Redis2-8-Bug记录一则/</id>
<published>2020-07-08T03:11:12.000Z</published>
<updated>2020-07-08T06:53:13.342Z</updated>
<content type="html"><![CDATA[<h1 id="INCR命令递增字符串错误"><a href="#INCR命令递增字符串错误" class="headerlink" title="INCR命令递增字符串错误"></a>INCR命令递增字符串错误</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">过程：</span><br><span class="line">127.0.0.1:6379> setrange string1 1 "Redis"</span><br><span class="line">(integer) 6</span><br><span class="line">127.0.0.1:6379> get string1</span><br><span class="line">"\x00Redis"</span><br><span class="line">127.0.0.1:6379> incr string1</span><br><span class="line">(integer) 1</span><br><span class="line">127.0.0.1:6379> get string1</span><br><span class="line">"1"</span><br></pre></td></tr></table></figure><p>经验证redis_version:4.0版本中，问题已经修复。<br>BUG修复PR：<a href="https://github.com/antirez/redis/commit/8b119999666f200685da61e4f0b1cfccc339f076" target="_blank" rel="noopener">https://github.com/antirez/redis/commit/8b119999666f200685da61e4f0b1cfccc339f076</a></p>]]></content>
<summary type="html">
<h1 id="INCR命令递增字符串错误"><a href="#INCR命令递增字符串错误" class="headerlink" title="INCR命令递增字符串错误"></a>INCR命令递增字符串错误</h1><figure class="highlight plai
</summary>
<category term="Redis" scheme="http://yoursite.com/categories/Redis/"/>
<category term="Redis" scheme="http://yoursite.com/tags/Redis/"/>
</entry>
<entry>
<title>Xtrabackup 恢复失败记录一则</title>
<link href="http://yoursite.com/2020/06/29/new/MySQL/Xtrabackup-%E6%81%A2%E5%A4%8D%E5%A4%B1%E8%B4%A5%E8%AE%B0%E5%BD%95%E4%B8%80%E5%88%99/"/>
<id>http://yoursite.com/2020/06/29/new/MySQL/Xtrabackup-恢复失败记录一则/</id>
<published>2020-06-29T12:14:57.000Z</published>
<updated>2020-06-29T12:15:32.422Z</updated>
<content type="html"><![CDATA[<h2 id="问题描述"><a href="#问题描述" class="headerlink" title="问题描述"></a>问题描述</h2><p>最近基于MySQL自动恢复脚本做集群实例功能，发现在部分机器上触发时会存在扩展失败的情况。<br>具体表现为：</p><ol><li>恢复失败机器为MySQL使用机器，此前是能够恢复成功。</li><li>通用备份机器仍然能够恢复成功。</li></ol><p>通过查看恢复日志发现报以下错误：<br><code>/bin/sh: line 1: 362302 Segmentation fault (core dumped)</code></p><p>注：MySQL自动恢复脚本是本人开发的一套自动备份恢复体系中的恢复模块，主要是利用<code>innobackupex</code>工具+传入参数能够进行一键化进行MySQL实例恢复操作。能够只恢复指定库表数据、指定GTID位点等。</p><h2 id="排查过程"><a href="#排查过程" class="headerlink" title="排查过程"></a>排查过程</h2><p>根据报错描述，段异常。感觉是在MySQL实例恢复过程中，访问了非法地址，类似于内存溢出场景。</p><p>第一想法是<code>innobackupex</code>二进制文件损坏，于是通过对比不同机器上<code>innobackupex</code>工具的版本以及md5值，发现并没有区别。所以排除工具问题。</p><p>接着，怀疑是机器某些配置被修改导致的，于是询问组内伙伴和OPS团队。均得到未修改的回答。。。。此处陷入没有头绪的情况。</p><p>但是我还是肯定是机器层面的问题，于是就开始了瞎猫碰死耗子的试探之路。不断的对比成功机器和失败机器之间的系统参数区别。<br>在<code>ulimit -a</code>时发现存在多个配置不同，于是通过一个一个修改尝试，最终确定是以下参数导致的：<br>失败机器：<code>stack size (kbytes, -s) 1024</code><br>成功机器：<code>stack size (kbytes, -s) 8192</code></p><p>问题原因确定，于是拿着这个参数询问。得到老大的回答，为了使DB机器配置标椎化，最近对MySQL机器的此参数进行了批量修改的操作。（此前Mongo机器都是1024）</p><h2 id="结论"><a href="#结论" class="headerlink" title="结论"></a>结论</h2><p>这个参数有什么用？为什么会导致MySQL恢复失败呢？</p><p>百度对<code>stack size</code>的表述是：进程的栈的最大值<br>在程序运行时，如果程序比较大，那么就会遇到“段错误”(segmentation fault)，主要就是收到了这个参数的限制。</p><p>PS：为什么不将这个值设置成默认值，而是调小呢？我老大说DB都是属于并发多的服务，这个值如果设置的太大的话，会导致存在大量无用内存的损耗。虽然没搞懂，我还是认为不应该修改这个值，当前太菜了，后面有机会再深入研究吧。</p>]]></content>
<summary type="html">
<h2 id="问题描述"><a href="#问题描述" class="headerlink" title="问题描述"></a>问题描述</h2><p>最近基于MySQL自动恢复脚本做集群实例功能，发现在部分机器上触发时会存在扩展失败的情况。<br>具体表现为：</p>
<o
</summary>
<category term="MySQL" scheme="http://yoursite.com/categories/MySQL/"/>
<category term="MySQL问题记录" scheme="http://yoursite.com/tags/MySQL%E9%97%AE%E9%A2%98%E8%AE%B0%E5%BD%95/"/>
</entry>
<entry>
<title>MySQL机制之排序算法</title>
<link href="http://yoursite.com/2020/06/27/MySQL%E6%9C%BA%E5%88%B6%E4%B9%8B%E6%8E%92%E5%BA%8F%E7%AE%97%E6%B3%95/"/>
<id>http://yoursite.com/2020/06/27/MySQL机制之排序算法/</id>
<published>2020-06-27T10:47:26.000Z</published>
<updated>2020-06-27T10:48:13.591Z</updated>
<content type="html"><![CDATA[<p>在很多场景中，我们都要求从数据库中查询出来的数据是有序的，SQL语法为<code>order by</code>。如果在查询使用到了排序，那么在EXPLAIN中，Extra 这个字段中会存在<code>Using filesort</code>。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">mysql> explain select city,name,age from t where city='杭州' order by name limit 1000 ;</span><br><span class="line">+----+-------------+-------+------------+------+---------------+------+---------+-------+------+----------+----------------+</span><br><span class="line">| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |</span><br><span class="line">+----+-------------+-------+------------+------+---------------+------+---------+-------+------+----------+----------------+</span><br><span class="line">| 1 | SIMPLE | t | NULL | ref | city | city | 66 | const | 1 | 100.00 | Using filesort |</span><br><span class="line">+----+-------------+-------+------------+------+---------------+------+---------+-------+------+----------+----------------+</span><br></pre></td></tr></table></figure></p><p>那么它在MySQL中是怎么实现的？什么参数会影响它呢？</p><h2 id="全字段排序"><a href="#全字段排序" class="headerlink" title="全字段排序"></a>全字段排序</h2><p>通常情况下，语句执行流程如下：</p><ol><li>初始化sort_buffer，确认需要放入的字段。(city,name,age)</li><li>从索引中查询满足条件的数据。</li><li>如果需要进行回表查询，那么会根据ID到聚簇索引中去查询到所需的字段。并将字段数据放入到sort_buffer中。</li><li>继续查询下一条记录，重复以上动作。直到将所有满足条件的数据都查询出来。</li><li>对sort_buffer中的数据按照排序字段进行排序。</li><li>将排序结果的前1000行返回给客户端。</li></ol><p><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gg70opqo9lj30vq0nsju1.jpg" alt=""></p><p>按照name排序的这个过程，可能在内存中完成，也可能需要使用外部排序。这取决于排序所需要的内存和参数<font color="red">sort_buffer_size</font></p><blockquote><p>sort_buffer_size，就是 MySQL 为排序开辟的内存（sort_buffer）的大小。如果要排序的数据量小于 sort_buffer_size，排序就在内存中完成。但如果排序数据量太大，内存放不下，则不得不利用磁盘临时文件辅助排序。</p></blockquote><a id="more"></a><h2 id="rowid排序"><a href="#rowid排序" class="headerlink" title="rowid排序"></a>rowid排序</h2><p>在上面的排序算法中，只对原表的数据读取了一遍，剩下的操作都是在sort_buffer和临时文件中执行。<br>但是，如果需要返回的字段很多的话，sort_buffer中需要保存所有的字段，此时sort_buffer中保存不了几条数据，需要使用很多个临时文件，排序效率太低了。</p><p>于是MySQL引入了<font color="red">max_length_for_sort_data</font>参数来进行控制，如果需要排序的字段总长度超过这个值，就会使用rowid排序算法。</p><blockquote><p>max_length_for_sort_data，是 MySQL 中专门控制用于排序的行数据的长度的一个参数。它的意思是，如果单行的长度超过这个值，MySQL 就认为单行太大，要换一个算法。</p></blockquote><p>在rowid排序中，整个排序过程就变成：</p><ol><li>初始化 sort_buffer，确定放入两个字段，即name和id</li><li>从索引中查找满足条件的数据的这两个字段值</li><li>取出这两个值保存到sort_buffer中</li><li>继续查询下一条记录，重复以上动作。直到将所有满足条件的数据都查询出来。</li><li>对sort_buffer中的数据进行排序</li><li>遍历排序结果，取前 1000 行，并按照 id 的值回到原表中取出 city、name 和 age 三个字段返回给客户端。</li></ol><p><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gg7110c6sij30vq0nsacz.jpg" alt=""></p><h2 id="优先队列排序"><a href="#优先队列排序" class="headerlink" title="优先队列排序"></a>优先队列排序</h2><p>MySQL 5.6 版本引入的一个新的排序算法，即：优先队列排序算法<br>在一个排序场景中，我们可能只需要前面几条的数据。如果采用的是上面的两种排序算法，那么需要对所有数据进行排序，这样造成了无谓的排序损耗。<br>于是MySQL引入了优先队列排序算法，只保存前面的几条数据信息。</p><p><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gg7153xo87j30u0140tf6.jpg" alt=""></p><h2 id="扩展-如何确认排序语句是否使用了临时文件"><a href="#扩展-如何确认排序语句是否使用了临时文件" class="headerlink" title="扩展:如何确认排序语句是否使用了临时文件"></a>扩展:如何确认排序语句是否使用了临时文件</h2><p>optimizer_trace 功能可以让我们方便的查看优化器生成执行计划的整个过程:<br>主要会体现以下阶段的具体计算：</p><ol><li>prepare阶段</li><li>optimize阶段</li><li>execute阶段</li><li>基于成本的优化主要集中在optimize阶段</li><li>单表查询来说，我们主要关注optimize阶段的”rows_estimation”这个过程，这个过程深入分析了对单表查询的各种执行方案的成本</li><li>对于多表连接查询来说，我们更多需要关注”considered_execution_plans”这个过程，这个过程里会写明各种不同的连接方式所对应的成本</li></ol><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">/* 打开optimizer_trace，只对本线程有效 */</span><br><span class="line">SET optimizer_trace='enabled=on'; </span><br><span class="line"></span><br><span class="line">/* @a保存Innodb_rows_read的初始值 */</span><br><span class="line">select VARIABLE_VALUE into @a from performance_schema.session_status where variable_name = 'Innodb_rows_read';</span><br><span class="line"></span><br><span class="line">/* 执行语句 */</span><br><span class="line">select city, name,age from t where city='杭州' order by name limit 1000; </span><br><span class="line"></span><br><span class="line">/* 查看 OPTIMIZER_TRACE 输出 */</span><br><span class="line">SELECT * FROM `information_schema`.`OPTIMIZER_TRACE`\G</span><br><span class="line"></span><br><span class="line">/* @b保存Innodb_rows_read的当前值 */</span><br><span class="line">select VARIABLE_VALUE into @b from performance_schema.session_status where variable_name = 'Innodb_rows_read';</span><br><span class="line"></span><br><span class="line">/* 计算Innodb_rows_read差值 */</span><br><span class="line">select @b-@a;</span><br></pre></td></tr></table></figure><p>optimizer_trace输出信息：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br><span class="line">201</span><br><span class="line">202</span><br><span class="line">203</span><br><span class="line">204</span><br><span class="line">205</span><br><span class="line">206</span><br><span class="line">207</span><br><span class="line">208</span><br><span class="line">209</span><br><span class="line">210</span><br><span class="line">211</span><br><span class="line">212</span><br><span class="line">213</span><br><span class="line">214</span><br><span class="line">215</span><br><span class="line">216</span><br><span class="line">217</span><br><span class="line">218</span><br><span class="line">219</span><br><span class="line">220</span><br><span class="line">221</span><br><span class="line">222</span><br><span class="line">223</span><br><span class="line">224</span><br><span class="line">225</span><br><span class="line">226</span><br><span class="line">227</span><br><span class="line">228</span><br><span class="line">229</span><br><span class="line">230</span><br><span class="line">231</span><br><span class="line">232</span><br><span class="line">233</span><br><span class="line">234</span><br><span class="line">235</span><br><span class="line">236</span><br><span class="line">237</span><br><span class="line">238</span><br><span class="line">239</span><br><span class="line">240</span><br><span class="line">241</span><br><span class="line">242</span><br><span class="line">243</span><br><span class="line">244</span><br><span class="line">245</span><br><span class="line">246</span><br><span class="line">247</span><br><span class="line">248</span><br><span class="line">249</span><br><span class="line">250</span><br><span class="line">251</span><br><span class="line">252</span><br><span class="line">253</span><br><span class="line">254</span><br><span class="line">255</span><br><span class="line">256</span><br><span class="line">257</span><br><span class="line">258</span><br><span class="line">259</span><br><span class="line">260</span><br><span class="line">261</span><br><span class="line">262</span><br><span class="line">263</span><br><span class="line">264</span><br></pre></td><td class="code"><pre><span class="line">mysql> SELECT * FROM `information_schema`.`OPTIMIZER_TRACE`\G</span><br><span class="line">*************************** 1. row ***************************</span><br><span class="line"> QUERY: select city, name,age from t where city='杭州' order by name limit 1000</span><br><span class="line"> TRACE: {</span><br><span class="line"> "steps": [</span><br><span class="line"> {</span><br><span class="line"> "join_preparation": {</span><br><span class="line"> "select#": 1,</span><br><span class="line"> "steps": [</span><br><span class="line"> {</span><br><span class="line"> "expanded_query": "/* select#1 */ select `t`.`city` AS `city`,`t`.`name` AS `name`,`t`.`age` AS `age` from `t` where (`t`.`city` = '杭州') order by `t`.`name` limit 1000"</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> "join_optimization": {</span><br><span class="line"> "select#": 1,</span><br><span class="line"> "steps": [</span><br><span class="line"> {</span><br><span class="line"> "condition_processing": {</span><br><span class="line"> "condition": "WHERE",</span><br><span class="line"> "original_condition": "(`t`.`city` = '杭州')",</span><br><span class="line"> "steps": [</span><br><span class="line"> {</span><br><span class="line"> "transformation": "equality_propagation",</span><br><span class="line"> "resulting_condition": "multiple equal('杭州', `t`.`city`)"</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> "transformation": "constant_propagation",</span><br><span class="line"> "resulting_condition": "multiple equal('杭州', `t`.`city`)"</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> "transformation": "trivial_condition_removal",</span><br><span class="line"> "resulting_condition": "multiple equal('杭州', `t`.`city`)"</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> "substitute_generated_columns": {</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> "table_dependencies": [</span><br><span class="line"> {</span><br><span class="line"> "table": "`t`",</span><br><span class="line"> "row_may_be_null": false,</span><br><span class="line"> "map_bit": 0,</span><br><span class="line"> "depends_on_map_bits": [</span><br><span class="line"> ]</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> "ref_optimizer_key_uses": [</span><br><span class="line"> {</span><br><span class="line"> "table": "`t`",</span><br><span class="line"> "field": "city",</span><br><span class="line"> "equals": "'杭州'",</span><br><span class="line"> "null_rejecting": false</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> "rows_estimation": [</span><br><span class="line"> {</span><br><span class="line"> "table": "`t`",</span><br><span class="line"> "range_analysis": {</span><br><span class="line"> "table_scan": {</span><br><span class="line"> "rows": 1,</span><br><span class="line"> "cost": 2.45</span><br><span class="line"> },</span><br><span class="line"> "potential_range_indexes": [</span><br><span class="line"> {</span><br><span class="line"> "index": "PRIMARY",</span><br><span class="line"> "usable": false,</span><br><span class="line"> "cause": "not_applicable"</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> "index": "city",</span><br><span class="line"> "usable": true,</span><br><span class="line"> "key_parts": [</span><br><span class="line"> "city",</span><br><span class="line"> "id"</span><br><span class="line"> ]</span><br><span class="line"> }</span><br><span class="line"> ],</span><br><span class="line"> "setup_range_conditions": [</span><br><span class="line"> ],</span><br><span class="line"> "group_index_range": {</span><br><span class="line"> "chosen": false,</span><br><span class="line"> "cause": "not_group_by_or_distinct"</span><br><span class="line"> },</span><br><span class="line"> "skip_scan_range": {</span><br><span class="line"> "potential_skip_scan_indexes": [</span><br><span class="line"> {</span><br><span class="line"> "index": "city",</span><br><span class="line"> "usable": false,</span><br><span class="line"> "cause": "query_references_nonkey_column"</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> },</span><br><span class="line"> "analyzing_range_alternatives": {</span><br><span class="line"> "range_scan_alternatives": [</span><br><span class="line"> {</span><br><span class="line"> "index": "city",</span><br><span class="line"> "ranges": [</span><br><span class="line"> "杭州 <= city <= 杭州"</span><br><span class="line"> ],</span><br><span class="line"> "index_dives_for_eq_ranges": true,</span><br><span class="line"> "rowid_ordered": true,</span><br><span class="line"> "using_mrr": false,</span><br><span class="line"> "index_only": false,</span><br><span class="line"> "rows": 1,</span><br><span class="line"> "cost": 0.61,</span><br><span class="line"> "chosen": true</span><br><span class="line"> }</span><br><span class="line"> ],</span><br><span class="line"> "analyzing_roworder_intersect": {</span><br><span class="line"> "usable": false,</span><br><span class="line"> "cause": "too_few_roworder_scans"</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> "chosen_range_access_summary": {</span><br><span class="line"> "range_access_plan": {</span><br><span class="line"> "type": "range_scan",</span><br><span class="line"> "index": "city",</span><br><span class="line"> "rows": 1,</span><br><span class="line"> "ranges": [</span><br><span class="line"> "杭州 <= city <= 杭州"</span><br><span class="line"> ]</span><br><span class="line"> },</span><br><span class="line"> "rows_for_plan": 1,</span><br><span class="line"> "cost_for_plan": 0.61,</span><br><span class="line"> "chosen": true</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> "considered_execution_plans": [</span><br><span class="line"> {</span><br><span class="line"> "plan_prefix": [</span><br><span class="line"> ],</span><br><span class="line"> "table": "`t`",</span><br><span class="line"> "best_access_path": {</span><br><span class="line"> "considered_access_paths": [</span><br><span class="line"> {</span><br><span class="line"> "access_type": "ref",</span><br><span class="line"> "index": "city",</span><br><span class="line"> "rows": 1,</span><br><span class="line"> "cost": 0.35,</span><br><span class="line"> "chosen": true</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> "access_type": "range",</span><br><span class="line"> "range_details": {</span><br><span class="line"> "used_index": "city"</span><br><span class="line"> },</span><br><span class="line"> "chosen": false,</span><br><span class="line"> "cause": "heuristic_index_cheaper"</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> },</span><br><span class="line"> "condition_filtering_pct": 100,</span><br><span class="line"> "rows_for_plan": 1,</span><br><span class="line"> "cost_for_plan": 0.35,</span><br><span class="line"> "chosen": true</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> "attaching_conditions_to_tables": {</span><br><span class="line"> "original_condition": "(`t`.`city` = '杭州')",</span><br><span class="line"> "attached_conditions_computation": [</span><br><span class="line"> ],</span><br><span class="line"> "attached_conditions_summary": [</span><br><span class="line"> {</span><br><span class="line"> "table": "`t`",</span><br><span class="line"> "attached": "(`t`.`city` = '杭州')"</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> "optimizing_distinct_group_by_order_by": {</span><br><span class="line"> "simplifying_order_by": {</span><br><span class="line"> "original_clause": "`t`.`name`",</span><br><span class="line"> "items": [</span><br><span class="line"> {</span><br><span class="line"> "item": "`t`.`name`"</span><br><span class="line"> }</span><br><span class="line"> ],</span><br><span class="line"> "resulting_clause_is_simple": true,</span><br><span class="line"> "resulting_clause": "`t`.`name`"</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> "finalizing_table_conditions": [</span><br><span class="line"> {</span><br><span class="line"> "table": "`t`",</span><br><span class="line"> "original_table_condition": "(`t`.`city` = '杭州')",</span><br><span class="line"> "final_table_condition ": null</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> "refine_plan": [</span><br><span class="line"> {</span><br><span class="line"> "table": "`t`"</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> "considering_tmp_tables": [</span><br><span class="line"> {</span><br><span class="line"> "adding_sort_to_table": "t"</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> "join_execution": {</span><br><span class="line"> "select#": 1,</span><br><span class="line"> "steps": [</span><br><span class="line"> {</span><br><span class="line"> "sorting_table": "t",</span><br><span class="line"> "filesort_information": [</span><br><span class="line"> {</span><br><span class="line"> "direction": "asc",</span><br><span class="line"> "expression": "`t`.`name`"</span><br><span class="line"> }</span><br><span class="line"> ],</span><br><span class="line"> "filesort_priority_queue_optimization": {</span><br><span class="line"> "limit": 1000</span><br><span class="line"> },</span><br><span class="line"> "filesort_execution": [</span><br><span class="line"> ],</span><br><span class="line"> "filesort_summary": {</span><br><span class="line"> "memory_available": 262144,</span><br><span class="line"> "key_size": 264,</span><br><span class="line"> "row_size": 406,</span><br><span class="line"> "max_rows_per_buffer": 633,</span><br><span class="line"> "num_rows_estimate": 1057,</span><br><span class="line"> "num_rows_found": 0,</span><br><span class="line"> "num_initial_chunks_spilled_to_disk": 0,</span><br><span class="line"> "peak_memory_used": 0,</span><br><span class="line"> "sort_algorithm": "none",</span><br><span class="line"> "sort_mode": "<varlen_sort_key, packed_additional_fields>"</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line">}</span><br><span class="line">MISSING_BYTES_BEYOND_MAX_MEM_SIZE: 0</span><br><span class="line"> INSUFFICIENT_PRIVILEGES: 0</span><br><span class="line">1 row in set (0.29 sec)</span><br></pre></td></tr></table></figure></p><blockquote><p>《MySQL实战45讲(16、17)》</p></blockquote>]]></content>
<summary type="html">
<p>在很多场景中，我们都要求从数据库中查询出来的数据是有序的，SQL语法为<code>order by</code>。如果在查询使用到了排序，那么在EXPLAIN中，Extra 这个字段中会存在<code>Using filesort</code>。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">mysql&gt; explain select city,name,age from t where city=&apos;杭州&apos; order by name limit 1000 ;</span><br><span class="line">+----+-------------+-------+------------+------+---------------+------+---------+-------+------+----------+----------------+</span><br><span class="line">| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |</span><br><span class="line">+----+-------------+-------+------------+------+---------------+------+---------+-------+------+----------+----------------+</span><br><span class="line">| 1 | SIMPLE | t | NULL | ref | city | city | 66 | const | 1 | 100.00 | Using filesort |</span><br><span class="line">+----+-------------+-------+------------+------+---------------+------+---------+-------+------+----------+----------------+</span><br></pre></td></tr></table></figure></p>
<p>那么它在MySQL中是怎么实现的？什么参数会影响它呢？</p>
<h2 id="全字段排序"><a href="#全字段排序" class="headerlink" title="全字段排序"></a>全字段排序</h2><p>通常情况下，语句执行流程如下：</p>
<ol>
<li>初始化sort_buffer，确认需要放入的字段。(city,name,age)</li>
<li>从索引中查询满足条件的数据。</li>
<li>如果需要进行回表查询，那么会根据ID到聚簇索引中去查询到所需的字段。并将字段数据放入到sort_buffer中。</li>
<li>继续查询下一条记录，重复以上动作。直到将所有满足条件的数据都查询出来。</li>
<li>对sort_buffer中的数据按照排序字段进行排序。</li>
<li>将排序结果的前1000行返回给客户端。</li>
</ol>
<p><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gg70opqo9lj30vq0nsju1.jpg" alt=""></p>
<p>按照name排序的这个过程，可能在内存中完成，也可能需要使用外部排序。这取决于排序所需要的内存和参数<font color="red">sort_buffer_size</font></p>
<blockquote>
<p>sort_buffer_size，就是 MySQL 为排序开辟的内存（sort_buffer）的大小。如果要排序的数据量小于 sort_buffer_size，排序就在内存中完成。但如果排序数据量太大，内存放不下，则不得不利用磁盘临时文件辅助排序。</p>
</blockquote>
</summary>
<category term="MySQL" scheme="http://yoursite.com/categories/MySQL/"/>
<category term="MySQL机制" scheme="http://yoursite.com/tags/MySQL%E6%9C%BA%E5%88%B6/"/>
</entry>
<entry>
<title>MySQL之MGR初涉</title>
<link href="http://yoursite.com/2020/06/27/new/MySQL/MySQL%E4%B9%8BMGR%E5%88%9D%E6%B6%89/"/>
<id>http://yoursite.com/2020/06/27/new/MySQL/MySQL之MGR初涉/</id>
<published>2020-06-27T10:00:20.000Z</published>
<updated>2020-06-27T10:01:21.061Z</updated>
<content type="html"><![CDATA[<h2 id="简介"><a href="#简介" class="headerlink" title="简介"></a>简介</h2><p>MySQL Group Replication（简称MGR）是MySQL官方于2016年12月推出的一个全新的高可用与高扩展的解决方案。MGR是MySQL官方在5.7.17版本引进的一个数据库高可用与高扩展的解决方案，以插件形式提供，实现了分布式下数据的最终一致性, 它是MySQL5.7版本出现的新特性，它提供了高可用、高扩展、高可靠的MySQL集群服务。组复制在数据库层面做到了高可用，只要集群中大多数主机可用，则服务可用。</p><h3 id="特点"><a href="#特点" class="headerlink" title="特点"></a>特点</h3><ul><li>高一致性：基于分布式paxos协议实现组复制，保证数据一致性。</li><li>高容错性：自动检测机制，内置防脑裂保护机。</li><li>高扩展性：节点增加移除自动更新组成员信息。新节点加入后，自动从其他节点同步增量数据。</li><li>高灵活性：提供单主和多主模式，单主宕机自动选主。</li></ul><h3 id="限制"><a href="#限制" class="headerlink" title="限制"></a>限制</h3><ul><li>存储引擎必须是InnoDB，即只支持InnoDB表</li><li>每张表都必须要有一个主键，用于做weite set的冲突检测</li><li>只支持ipv4，网络需求较高</li><li>必须开启GTID</li><li>binlog格式必须设置为ROW</li><li>一个MGR集群最多支持9个节点</li><li>不支持外键于save point特性，无法做全局间的约束检测与部分部分回滚</li><li>二进制日志binlog不支持Replication event checksums</li><li>多主模式(也就是多写模式) 不支持SERIALIZABLE事务隔离级别</li><li>多主模式不能完全支持级联外键约束</li><li>多主模式不支持在不同节点上对同一个数据库对象并发执行DDL(在不同节点上对同一行并发进行RW事务，后发起的事务会失败)</li></ul><p>搭建过程可参考<a href="https://omg-by.github.io/2020/06/13/new/MySQL/Docker-%E7%8E%AF%E5%A2%83%E4%B8%8B%E6%90%AD%E5%BB%BAMySQL-8-0-20-%E7%BB%84%E5%A4%8D%E5%88%B6/" target="_blank" rel="noopener">Docker 环境下搭建MySQL 8.0.20 组复制</a></p><a id="more"></a><h2 id="单主模式和多主模式"><a href="#单主模式和多主模式" class="headerlink" title="单主模式和多主模式"></a>单主模式和多主模式</h2><p>MySQL组复制分单主模式和多主模式。</p><p>在<code>单主模式</code>下，组复制具有自动选主功能，每次只有一个server成员接受更新。单主模式下，group内只有一个节点可读可写，其他节点只可以读。当primary节点意外宕机或下线，在集群仍满足大多数原则情况下，group内部发起选举，选出一个可用的读节点，提升为primary节点。primary选举根据group内剩下的存活节点的UUID按字典序升序选择，即剩余存活节点按UUID字典序排序，然后选择排在最前面的节点作为新的primary节点。</p><p>在<code>多主模式</code>下，所有server成员都可以同时接受更新操作，group内的所有机器都是primary节点，同时可以进行读写操作，并且数据最终一致性。</p><h2 id="MGR集群中事务的生命周期"><a href="#MGR集群中事务的生命周期" class="headerlink" title="MGR集群中事务的生命周期"></a>MGR集群中事务的生命周期</h2><p><img src="https://tva1.sinaimg.cn/large/007S8ZIlly1gfst1lyqojj30e60c1wfh.jpg" alt=""></p><center>MGR架构示意图</center><p><img src="https://tva1.sinaimg.cn/large/007S8ZIlly1gfst7qqejcj30go088aaw.jpg" alt=""></p><p>如图，DB1、DB2、DB3构成的MGR集群中，每个DB都有MGR层，MGR层功能可以简单理解为由Paxos模块和冲突检测Certify模块实现。</p><p>当 DB1 上有事务 T1 要执行时，T1 对 DB1 是来说本地事务，对于 DB2、DB3 来说是远端事务;DB1 上在事务 T1 在被执行后，会把执行事务 T1 信息广播给集群各个节点，包括 DB1 本身，通过 Paxos 模块广播给 MGR 集群各个节点，半数以上的节点同意并且达成共识，之后共识信息进入各个节点的冲突检测 certify 模块，各个节点各自进行冲突检测验证，最终保证事务在集群中最终一致性。</p><p>在冲突检测通过之后，本地事务 T1 在 DB1 直接提交即可，否则直接回滚。远端事务 T1 在 DB2 和 DB3 分别先更新到 relay log，然后应用到 binlog，完成数据的同步，否则直接放弃该事务。</p><blockquote><p><a href="https://yq.aliyun.com/articles/640117" target="_blank" rel="noopener">https://yq.aliyun.com/articles/640117</a><br><a href="https://database.51cto.com/art/202004/615706.htm?mobile" target="_blank" rel="noopener">https://database.51cto.com/art/202004/615706.htm?mobile</a><br><a href="https://www.cnblogs.com/williamzheng/p/11347362.html" target="_blank" rel="noopener">https://www.cnblogs.com/williamzheng/p/11347362.html</a><br><a href="https://www.cnblogs.com/zzdbullet/p/11655434.html" target="_blank" rel="noopener">https://www.cnblogs.com/zzdbullet/p/11655434.html</a><br><a href="https://www.cnblogs.com/fangxuanlang/p/10383544.html" target="_blank" rel="noopener">https://www.cnblogs.com/fangxuanlang/p/10383544.html</a></p></blockquote>]]></content>
<summary type="html">
<h2 id="简介"><a href="#简介" class="headerlink" title="简介"></a>简介</h2><p>MySQL Group Replication（简称MGR）是MySQL官方于2016年12月推出的一个全新的高可用与高扩展的解决方案。MGR是MySQL官方在5.7.17版本引进的一个数据库高可用与高扩展的解决方案，以插件形式提供，实现了分布式下数据的最终一致性, 它是MySQL5.7版本出现的新特性，它提供了高可用、高扩展、高可靠的MySQL集群服务。组复制在数据库层面做到了高可用，只要集群中大多数主机可用，则服务可用。</p>
<h3 id="特点"><a href="#特点" class="headerlink" title="特点"></a>特点</h3><ul>
<li>高一致性：基于分布式paxos协议实现组复制，保证数据一致性。</li>
<li>高容错性：自动检测机制，内置防脑裂保护机。</li>
<li>高扩展性：节点增加移除自动更新组成员信息。新节点加入后，自动从其他节点同步增量数据。</li>
<li>高灵活性：提供单主和多主模式，单主宕机自动选主。</li>
</ul>
<h3 id="限制"><a href="#限制" class="headerlink" title="限制"></a>限制</h3><ul>
<li>存储引擎必须是InnoDB，即只支持InnoDB表</li>
<li>每张表都必须要有一个主键，用于做weite set的冲突检测</li>
<li>只支持ipv4，网络需求较高</li>
<li>必须开启GTID</li>
<li>binlog格式必须设置为ROW</li>
<li>一个MGR集群最多支持9个节点</li>
<li>不支持外键于save point特性，无法做全局间的约束检测与部分部分回滚</li>
<li>二进制日志binlog不支持Replication event checksums</li>
<li>多主模式(也就是多写模式) 不支持SERIALIZABLE事务隔离级别</li>
<li>多主模式不能完全支持级联外键约束</li>
<li>多主模式不支持在不同节点上对同一个数据库对象并发执行DDL(在不同节点上对同一行并发进行RW事务，后发起的事务会失败)</li>
</ul>
<p>搭建过程可参考<a href="https://omg-by.github.io/2020/06/13/new/MySQL/Docker-%E7%8E%AF%E5%A2%83%E4%B8%8B%E6%90%AD%E5%BB%BAMySQL-8-0-20-%E7%BB%84%E5%A4%8D%E5%88%B6/" target="_blank" rel="noopener">Docker 环境下搭建MySQL 8.0.20 组复制</a></p>
</summary>
<category term="MySQL" scheme="http://yoursite.com/categories/MySQL/"/>
<category term="MySQL" scheme="http://yoursite.com/tags/MySQL/"/>
</entry>
<entry>
<title>Go学习之Module介绍</title>
<link href="http://yoursite.com/2020/06/26/new/Go/Go%E5%AD%A6%E4%B9%A0%E4%B9%8BModule%E4%BB%8B%E7%BB%8D/"/>
<id>http://yoursite.com/2020/06/26/new/Go/Go学习之Module介绍/</id>
<published>2020-06-26T15:30:36.000Z</published>
<updated>2020-06-26T15:32:33.270Z</updated>
<content type="html"><![CDATA[<h2 id="最古老的GOPATH"><a href="#最古老的GOPATH" class="headerlink" title="最古老的GOPATH"></a>最古老的GOPATH</h2><p>Go的包管理方式是逐渐演进的，最初所有的包都放在GOPATH中，使用类似命名空间的包路径来区分包。<br>你可以将其理解为工作目录，在这个工作目录下，通常有如下的目录结构：</p><ul><li>bin：存放编译生成的二进制文件</li><li>pkg：存放编译后的.a文件</li><li>src：存放项目的源代码，可以是自己写的代码。也可以是go get下载的包</li></ul><p>在这种模式下，最严重的问题就是<font color="red">版本管理问题</font>，因为GOPATH没有版本的概念，所以可能会遇到以下的问题：</p><ul><li>无法在项目中使用指定版本的包，因为不同版本的包的导入方法是一样的。</li><li>其他人运行你开发的程序时，无法保证他下载的包版本是你所期望的版本。当其他人</li><li>本地中，一个包只能保留一个版本，意味着你在本地开发的所有项目都是一个版本的包</li></ul><a id="more"></a><h2 id="vendor模式"><a href="#vendor模式" class="headerlink" title="vendor模式"></a>vendor模式</h2><p>为了解决GOPATH方案下不同项目下无法使用多个版本库的问题，GO1.5开始支持vendor。</p><p>以前使用GOPATH的时候，所有项目都共享一个GOPATH，需要导入依赖的时候，都来这里找。在GOPATH模式下，第三方库都只能有一个版本。</p><p>解决的思路是：在每个项目下都创建一个vendor目录，每个项目所需要的依赖都只会下载到自己的vendor目录下，项目之间的依赖包互不影响。在编译时，v1.5的GO在设置了开启<code>GO15VENDOREXPERIMENT=1</code>。<br>（注：这个变量在 v1.6 版本默认为1，但是在 v1.7 后，已去掉该环境变量，默认开启 vendor 特性，无需你手动设置）后，会提升 vendor 目录的依赖包搜索路径的优先级（相较于 GOPATH）。</p><p>其搜索包的优先顺序为：</p><ul><li>当前包下的 vendor 目录</li><li>向上级目录查找，直到找到 src 下的 vendor 目录</li><li>在 GOROOT 目录下查找</li><li>在 GOPATH 下面查找依赖包</li></ul><p>虽然这个方案解决了一些问题，但是解决得并不完美。</p><ul><li>如果多个项目用到了同一个包的同一个版本，这个包会存在于该机器上的不同目录下，不仅对磁盘空间是一种浪费，而且没法对第三方包进行集中式的管理（分散在各个角落）。</li><li>并且如果要分享开源你的项目，你需要将你的所有的依赖包悉数上传，别人使用的时候，除了你的项目源码外，还有所有的依赖包全部下载下来，才能保证别人使用的时候，不会因为版本问题导致项目不能如你预期那样正常运行。</li></ul><h2 id="Go-Modules"><a href="#Go-Modules" class="headerlink" title="Go Modules"></a>Go Modules</h2><p>Go Modules在v1.11版本正式推出，在v1.14版本中，官方正式发话，称其已经足够成熟，可以应用于生产上。</p><p>从 v1.11 开始，go env 多了个环境变量：GO111MODULE ，这里的 111，其实就是 v1.11 的象征标志， go 里好像很喜欢这样的命名方式，比如当初 vendor 出现的时候，也多了个 GO15VENDOREXPERIMENT环境变量，其中 15，表示的vendor 是在 v1.5 时才诞生的。</p><p>GO111MODULE 是一个开关，通过它可以开启或关闭 go mod 模式。它有三个可选值：off、on、auto，默认值是auto。</p><ul><li>GO111MODULE=off禁用模块支持，编译时会从GOPATH和vendor文件夹中查找包。</li><li>GO111MODULE=on启用模块支持，编译时会忽略GOPATH和vendor文件夹，只根据 go.mod下载依赖。</li><li>GO111MODULE=auto，当项目在$GOPATH/src外且项目根目录有go.mod文件时，自动开启模块支持。</li></ul><h3 id="go-mod"><a href="#go-mod" class="headerlink" title="go mod"></a>go mod</h3><p>golang提供<code>go mod</code>命令来管理Modules包<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line">➜ test go mod</span><br><span class="line">Go mod provides access to operations on modules.</span><br><span class="line"></span><br><span class="line">Note that support for modules is built into all the go commands,</span><br><span class="line">not just 'go mod'. For example, day-to-day adding, removing, upgrading,</span><br><span class="line">and downgrading of dependencies should be done using 'go get'.</span><br><span class="line">See 'go help modules' for an overview of module functionality.</span><br><span class="line"></span><br><span class="line">Usage:</span><br><span class="line"></span><br><span class="line">go mod <command> [arguments]</span><br><span class="line"></span><br><span class="line">The commands are:</span><br><span class="line"></span><br><span class="line">download download modules to local cache</span><br><span class="line">edit edit go.mod from tools or scripts</span><br><span class="line">graph print module requirement graph</span><br><span class="line">init initialize new module in current directory</span><br><span class="line">tidy add missing and remove unused modules</span><br><span class="line">vendor make vendored copy of dependencies</span><br><span class="line">verify verify dependencies have expected content</span><br><span class="line">why explain why packages or modules are needed</span><br></pre></td></tr></table></figure></p><table><thead><tr><th style="text-align:center">命令</th><th style="text-align:center">说明</th></tr></thead><tbody><tr><td style="text-align:center">download</td><td style="text-align:center">下载依赖包</td></tr><tr><td style="text-align:center">edit</td><td style="text-align:center">下载依赖包</td></tr><tr><td style="text-align:center">graph</td><td style="text-align:center">编辑go.mod</td></tr><tr><td style="text-align:center">init</td><td style="text-align:center">打印模块依赖图</td></tr><tr><td style="text-align:center">tidy</td><td style="text-align:center">在当前目录初始化mod</td></tr><tr><td style="text-align:center">vendor</td><td style="text-align:center">拉取缺少的模块，移除不用的模块</td></tr><tr><td style="text-align:center">verify</td><td style="text-align:center">验证依赖是否正确</td></tr><tr><td style="text-align:center">why</td><td style="text-align:center">解释为什么需要依赖</td></tr></tbody></table><h3 id="使用"><a href="#使用" class="headerlink" title="使用"></a>使用</h3><h4 id="创建一个新项目"><a href="#创建一个新项目" class="headerlink" title="创建一个新项目"></a>创建一个新项目</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">➜ test go mod init test</span><br><span class="line">go: creating new go.mod: module test</span><br><span class="line">➜ test ls</span><br><span class="line">go.mod</span><br><span class="line">➜ test cat go.mod</span><br><span class="line">module test</span><br></pre></td></tr></table></figure><p>go.mod文件一旦创建后，它的内容将会被go toolchain全面掌控。go toolchain会在各类命令执行时，比如go get、go build、go mod等修改和维护go.mod文件。</p><p>go.mod 提供了以下四个命令：</p><ul><li>module 语句指定包的名字（路径）</li><li>require 语句指定的依赖项模块</li><li>replace 语句可以替换依赖项模块</li><li>exclude 语句可以忽略依赖项模块</li></ul><h4 id="添加依赖"><a href="#添加依赖" class="headerlink" title="添加依赖"></a>添加依赖</h4><p>新建一个 server.go 文件<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">package main</span><br><span class="line"></span><br><span class="line">import (</span><br><span class="line">"net/http"</span><br><span class="line"></span><br><span class="line">"github.com/labstack/echo"</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line">func main() {</span><br><span class="line">e := echo.New()</span><br><span class="line">e.GET("/", func(c echo.Context) error {</span><br><span class="line">return c.String(http.StatusOK, "Hello, World!")</span><br><span class="line">})</span><br><span class="line">e.Logger.Fatal(e.Start(":1323"))</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>执行<code>go run server.go</code>运行代码会发现 go mod 会自动查找依赖自动下载<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line">➜ test go run server.go</span><br><span class="line">go: finding github.com/labstack/echo v3.3.10+incompatible</span><br><span class="line">go: downloading github.com/labstack/echo v3.3.10+incompatible</span><br><span class="line">go: finding github.com/labstack/gommon/color latest</span><br><span class="line">go: finding github.com/labstack/gommon/log latest</span><br><span class="line">go: finding github.com/labstack/gommon v0.3.0</span><br><span class="line">go: downloading github.com/labstack/gommon v0.3.0</span><br><span class="line">go: finding golang.org/x/crypto/acme/autocert latest</span><br><span class="line">go: finding golang.org/x/crypto/acme latest</span><br><span class="line">go: finding golang.org/x/crypto latest</span><br><span class="line">go: downloading golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9</span><br><span class="line">go: finding github.com/mattn/go-isatty v0.0.9</span><br><span class="line">go: finding github.com/mattn/go-colorable v0.1.2</span><br><span class="line">go: finding github.com/valyala/fasttemplate v1.0.1</span><br><span class="line">go: finding github.com/mattn/go-isatty v0.0.8</span><br><span class="line">go: finding golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a</span><br><span class="line">go: finding github.com/valyala/bytebufferpool v1.0.0</span><br><span class="line">go: finding golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223</span><br><span class="line">go: downloading github.com/mattn/go-isatty v0.0.9</span><br><span class="line">go: downloading github.com/mattn/go-colorable v0.1.2</span><br><span class="line">go: downloading github.com/valyala/fasttemplate v1.0.1</span><br><span class="line">go: downloading golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3</span><br><span class="line">go: downloading github.com/valyala/bytebufferpool v1.0.0</span><br></pre></td></tr></table></figure></p><p>查看目录，发现多了<code>go.sum</code>文件。该文件主要用来记录 dependency tree。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">➜ test ls</span><br><span class="line">go.mod go.sum server.go</span><br><span class="line">➜ test cat go.mod</span><br><span class="line">module test</span><br><span class="line"></span><br><span class="line">require (</span><br><span class="line">github.com/labstack/echo v3.3.10+incompatible // indirect</span><br><span class="line">github.com/labstack/gommon v0.3.0 // indirect</span><br><span class="line">golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 // indirect</span><br><span class="line">)</span><br><span class="line">➜ test cat go go.sum</span><br><span class="line">cat: go: No such file or directory</span><br><span class="line">github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=</span><br><span class="line">github.com/labstack/echo v3.3.10+incompatible h1:pGRcYk231ExFAyoAjAfD85kQzRJCRI8bbnE7CX5OEgg=</span><br><span class="line">github.com/labstack/echo v3.3.10+incompatible/go.mod h1:0INS7j/VjnFxD4E2wkz67b8cVwCLbBmJyDaka6Cmk1s=</span><br><span class="line">github.com/labstack/gommon v0.3.0 h1:JEeO0bvc78PKdyHxloTKiF8BD5iGrH8T6MSeGvSgob0=</span><br><span class="line">github.com/labstack/gommon v0.3.0/go.mod h1:MULnywXg0yavhxWKc+lOruYdAhDwPK9wf0OL7NoOu+k=</span><br><span class="line">...</span><br></pre></td></tr></table></figure></p><p>当依赖都存在时，就会跳过下载的步骤。<br>可以使用命令<code>go list -m -u all</code>来检查可以升级的package，使用<code>go get -u need-upgrade-package</code>升级后会将新的依赖版本更新到go.mod * 也可以使用<code>go get -u</code>升级所有依赖</p><p>现在很少用GO，所以也没了解更深，这篇文章作为记录吧。剩下的后面在使用过程中再补充。。。。。</p>]]></content>
<summary type="html">
<h2 id="最古老的GOPATH"><a href="#最古老的GOPATH" class="headerlink" title="最古老的GOPATH"></a>最古老的GOPATH</h2><p>Go的包管理方式是逐渐演进的，最初所有的包都放在GOPATH中，使用类似命名空间的包路径来区分包。<br>你可以将其理解为工作目录，在这个工作目录下，通常有如下的目录结构：</p>
<ul>
<li>bin：存放编译生成的二进制文件</li>
<li>pkg：存放编译后的.a文件</li>
<li>src：存放项目的源代码，可以是自己写的代码。也可以是go get下载的包</li>
</ul>
<p>在这种模式下，最严重的问题就是<font color="red">版本管理问题</font>，因为GOPATH没有版本的概念，所以可能会遇到以下的问题：</p>
<ul>
<li>无法在项目中使用指定版本的包，因为不同版本的包的导入方法是一样的。</li>
<li>其他人运行你开发的程序时，无法保证他下载的包版本是你所期望的版本。当其他人</li>
<li>本地中，一个包只能保留一个版本，意味着你在本地开发的所有项目都是一个版本的包</li>
</ul>
</summary>
<category term="GO" scheme="http://yoursite.com/categories/GO/"/>
<category term="GO" scheme="http://yoursite.com/tags/GO/"/>
</entry>
<entry>
<title>Docker 环境下搭建MySQL 8.0.20 组复制</title>
<link href="http://yoursite.com/2020/06/13/new/MySQL/Docker-%E7%8E%AF%E5%A2%83%E4%B8%8B%E6%90%AD%E5%BB%BAMySQL-8-0-20-%E7%BB%84%E5%A4%8D%E5%88%B6/"/>
<id>http://yoursite.com/2020/06/13/new/MySQL/Docker-环境下搭建MySQL-8-0-20-组复制/</id>
<published>2020-06-12T16:57:03.000Z</published>
<updated>2020-07-13T02:31:58.515Z</updated>
<content type="html"><![CDATA[<p>最近在学习MySQL 8.0 新特性，以及MGR。于是想在搭建一套环境来测试。<br>本来在买的阿里云服务器上进行搭建，结果好像是一台机器搭建不了，至少需要三台，于是就放弃了。<br>正好小伙伴分享了docker，于是就利用docker在本地搭建了一套8.0的MGR环境。</p><h2 id="准备docker镜像"><a href="#准备docker镜像" class="headerlink" title="准备docker镜像"></a>准备docker镜像</h2><p>运行一个test名称的centos系统镜像<br><code>➜ ~ docker run -d -it --name test centos</code><br>进入容器<br><code>➜ ~ docker exec -it test bash</code><br>在容器内安装MySQL<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[root@c610b533f3dc /]# yum install yum-utils wget -y</span><br><span class="line">[root@c610b533f3dc /]# wget https://dev.mysql.com/get/mysql80-community-release-el7-1.noarch.rpm</span><br><span class="line">[root@c610b533f3dc /]# rpm -ivh mysql80-community-release-el7-1.noarch.rpm</span><br><span class="line">[root@c610b533f3dc /]# yum install -y mysql-server</span><br></pre></td></tr></table></figure></p><p>安装完后退出容器，并生成镜像<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@c610b533f3dc /]# exit</span><br><span class="line">➜ ~ docker commit 容器ID mysql8.0.20</span><br></pre></td></tr></table></figure></p><h2 id="启动MySQL容器"><a href="#启动MySQL容器" class="headerlink" title="启动MySQL容器"></a>启动MySQL容器</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">docker run -d -it --privileged --name=mysql-mgr-node1 mysql8.0.20 /usr/sbin/init</span><br><span class="line">docker run -d -it --privileged --name=mysql-mgr-node2 mysql8.0.20 /usr/sbin/init</span><br><span class="line">docker run -d -it --privileged --name=mysql-mgr-node3 mysql8.0.20 /usr/sbin/init</span><br></pre></td></tr></table></figure><a id="more"></a><h2 id="配置MGR节点"><a href="#配置MGR节点" class="headerlink" title="配置MGR节点"></a>配置MGR节点</h2><h3 id="修改-etc-hosts文件"><a href="#修改-etc-hosts文件" class="headerlink" title="修改/etc/hosts文件"></a>修改/etc/hosts文件</h3><p>登录每一个容器，查看hosts文件，并进行添加<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">➜ ~ docker exec -it mysql-mgr-node1 bash</span><br><span class="line">[root@4af6073b5ee9 /]# cat /etc/hosts</span><br><span class="line">127.0.0.1localhost</span><br><span class="line">::1localhost ip6-localhost ip6-loopback</span><br><span class="line">fe00::0ip6-localnet</span><br><span class="line">ff00::0ip6-mcastprefix</span><br><span class="line">ff02::1ip6-allnodes</span><br><span class="line">ff02::2ip6-allrouters</span><br><span class="line">172.17.0.34af6073b5ee9 </span><br><span class="line">172.17.0.4762f16c6d69f</span><br><span class="line">172.17.0.5927681f0afc0</span><br></pre></td></tr></table></figure></p><h3 id="修改-etc-my-cnf文件"><a href="#修改-etc-my-cnf文件" class="headerlink" title="修改/etc/my.cnf文件"></a>修改/etc/my.cnf文件</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">[mysqld]</span><br><span class="line">datadir=/var/lib/mysql</span><br><span class="line">socket=/var/lib/mysql/mysql.sock</span><br><span class="line"></span><br><span class="line">log-error=/var/log/mysqld.log</span><br><span class="line">pid-file=/var/run/mysqld/mysqld.pid</span><br><span class="line"></span><br><span class="line">server_id = 1</span><br><span class="line">gtid_mode = ON</span><br><span class="line">enforce_gtid_consistency = ON</span><br><span class="line">binlog_checksum = NONE</span><br><span class="line"></span><br><span class="line">transaction_write_set_extraction = XXHASH64</span><br><span class="line">loose-group_replication_group_name = 'ce9be252-2b71-11e6-b8f4-00212844f856'</span><br><span class="line">loose-group_replication_start_on_boot = off</span><br><span class="line">loose-group_replication_local_address= "172.17.0.3:33061"</span><br><span class="line">loose-group_replication_group_seeds= "172.17.0.3:33061,172.17.0.4:33061,172.17.0.5:33061"</span><br><span class="line">loose-group_replication_bootstrap_group = off</span><br></pre></td></tr></table></figure><p>其他两个容器只需要修改<code>server_id</code>和<code>loose-group_replication_local_address</code></p><h3 id="重启MySQL"><a href="#重启MySQL" class="headerlink" title="重启MySQL"></a>重启MySQL</h3><p><code>[root@4af6073b5ee9 /]# systemctl restart mysqld</code></p><h3 id="修改MySQL密码"><a href="#修改MySQL密码" class="headerlink" title="修改MySQL密码"></a>修改MySQL密码</h3><p>在容器的/var/log/mysqld.log文件中会有MySQL的初始化密码。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@4af6073b5ee9 /]# cat /var/log/mysqld.log</span><br><span class="line">2020-06-12T10:15:04.837913Z 6 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: gYEdgjDqe0,9</span><br></pre></td></tr></table></figure></p><p>登录MySQL<br><code>[root@4af6073b5ee9 /]# mysql -u root -p"gYEdgjDqe0,9"</code></p><p>修改密码<br><code>mysql> alter user 'root'@'localhost' identified by 'xxxxx';</code></p><h3 id="初始化组复制插件及用户"><a href="#初始化组复制插件及用户" class="headerlink" title="初始化组复制插件及用户"></a>初始化组复制插件及用户</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">mysql> INSTALL PLUGIN group_replication SONAME 'group_replication.so';</span><br><span class="line">mysql> SET SQL_LOG_BIN=0;</span><br><span class="line">mysql> CREATE USER repl@'%' IDENTIFIED BY 'xxxx';</span><br><span class="line">mysql> GRANT REPLICATION SLAVE ON *.* TO repl@'%';</span><br><span class="line">mysql> FLUSH PRIVILEGES;</span><br><span class="line">mysql> SET SQL_LOG_BIN=1;</span><br><span class="line">mysql> CHANGE MASTER TO MASTER_USER='repl', MASTER_PASSWORD='xxxx'FOR CHANNEL 'group_replication_recovery';</span><br></pre></td></tr></table></figure><font color="red">以上操作需要在所有容器上执行</font><h3 id="配置MGR主节点"><a href="#配置MGR主节点" class="headerlink" title="配置MGR主节点"></a>配置MGR主节点</h3><p>本操作在主节点(mysql-mgr-node1)上执行<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">mysql> SET GLOBAL group_replication_bootstrap_group=ON;</span><br><span class="line">mysql> START GROUP_REPLICATION;</span><br><span class="line">mysql> SET GLOBAL group_replication_bootstrap_group=OFF;</span><br><span class="line"></span><br><span class="line"># 查看组信息</span><br><span class="line">mysql> SELECT * FROM performance_schema.replication_group_members;</span><br><span class="line">+---------------------------+--------------------------------------+--------------+-------------+--------------+-------------+----------------+</span><br><span class="line">| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |</span><br><span class="line">+---------------------------+--------------------------------------+--------------+-------------+--------------+-------------+----------------+</span><br><span class="line">| group_replication_applier | 95744fa5-ac95-11ea-a21d-0242ac110003 | 4af6073b5ee9 | 3306 | ONLINE | PRIMARY | 8.0.20 |</span><br><span class="line">+---------------------------+--------------------------------------+--------------+-------------+--------------+-------------+----------------+</span><br></pre></td></tr></table></figure></p><h3 id="其他节点加入集群"><a href="#其他节点加入集群" class="headerlink" title="其他节点加入集群"></a>其他节点加入集群</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mysql> START GROUP_REPLICATION;</span><br></pre></td></tr></table></figure><p>在任意节点上可以查看组复制状态<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">mysql> SELECT * FROM performance_schema.replication_group_members;</span><br><span class="line">+---------------------------+--------------------------------------+--------------+-------------+--------------+-------------+----------------+</span><br><span class="line">| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |</span><br><span class="line">+---------------------------+--------------------------------------+--------------+-------------+--------------+-------------+----------------+</span><br><span class="line">| group_replication_applier | 95744fa5-ac95-11ea-a21d-0242ac110003 | 4af6073b5ee9 | 3306 | ONLINE | PRIMARY | 8.0.20 |</span><br><span class="line">| group_replication_applier | 9bda89f2-ac95-11ea-9c6a-0242ac110004 | 762f16c6d69f | 3306 | ONLINE | SECONDARY | 8.0.20 |</span><br><span class="line">| group_replication_applier | 9f569a5e-ac95-11ea-9ffc-0242ac110005 | 927681f0afc0 | 3306 | ONLINE | SECONDARY | 8.0.20 |</span><br><span class="line">+---------------------------+--------------------------------------+--------------+-------------+--------------+-------------+----------------+</span><br></pre></td></tr></table></figure></p><p>至此，在Docker下搭建MySQL MGR完成</p><h3 id="MGR多主模式搭建-扩展"><a href="#MGR多主模式搭建-扩展" class="headerlink" title="MGR多主模式搭建(扩展)"></a>MGR多主模式搭建(扩展)</h3><p>在所有MGR节点上停止组复制模式，并配置多主模式参数<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">mysql> stop group_replication;</span><br><span class="line">mysql> set global group_replication_single_primary_mode=OFF;</span><br><span class="line">mysql> set global group_replication_enforce_update_everywhere_checks=ON;</span><br></pre></td></tr></table></figure></p><p>选一个节点作为源节点，执行以下命令<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">mysql> SET GLOBAL group_replication_bootstrap_group=ON;</span><br><span class="line">mysql> START GROUP_REPLICATION;</span><br><span class="line">mysql> SET GLOBAL group_replication_bootstrap_group=OFF;</span><br></pre></td></tr></table></figure></p><p>其他节点加入集群<br><code>mysql> START GROUP_REPLICATION;</code></p><p>查看MGR组状态<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">mysql> SELECT * FROM performance_schema.replication_group_members;</span><br><span class="line">+---------------------------+--------------------------------------+--------------+-------------+--------------+-------------+----------------+</span><br><span class="line">| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |</span><br><span class="line">+---------------------------+--------------------------------------+--------------+-------------+--------------+-------------+----------------+</span><br><span class="line">| group_replication_applier | 95744fa5-ac95-11ea-a21d-0242ac110003 | 4af6073b5ee9 | 3306 | ONLINE | PRIMARY | 8.0.20 |</span><br><span class="line">| group_replication_applier | 9bda89f2-ac95-11ea-9c6a-0242ac110004 | 762f16c6d69f | 3306 | ONLINE | PRIMARY | 8.0.20 |</span><br><span class="line">| group_replication_applier | 9f569a5e-ac95-11ea-9ffc-0242ac110005 | 927681f0afc0 | 3306 | ONLINE | PRIMARY | 8.0.20 |</span><br><span class="line">+---------------------------+--------------------------------------+--------------+-------------+--------------+-------------+----------------+</span><br></pre></td></tr></table></figure></p><h2 id="遇到的问题"><a href="#遇到的问题" class="headerlink" title="遇到的问题"></a>遇到的问题</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[ERROR] Slave I/O for channel 'group_replication_recovery': Got fatal error 1236 from master when</span><br><span class="line">reading data from binary log: 'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has</span><br><span class="line">purged binary logs containing GTIDs that the slave requires.', Error_code: 1236</span><br></pre></td></tr></table></figure><p>这是由于主从实例上都产生了gtid，导致冲突报错。<br>解决办法：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"># 在主实例上执行show master status\G命令查看gtid</span><br><span class="line">mysql> show master status\G</span><br><span class="line">*************************** 1. row ***************************</span><br><span class="line"> File: binlog.000002</span><br><span class="line"> Position: 1365</span><br><span class="line"> Binlog_Do_DB:</span><br><span class="line"> Binlog_Ignore_DB:</span><br><span class="line">Executed_Gtid_Set: 95744fa5-ac95-11ea-a21d-0242ac110003:1-2,</span><br><span class="line">ce9be252-2b71-11e6-b8f4-00212844f856:1-2</span><br><span class="line"></span><br><span class="line"># 在从实例上重新设定gtid</span><br><span class="line">mysql> STOP GROUP_REPLICATION;</span><br><span class="line">mysql> reset master; </span><br><span class="line"></span><br><span class="line"># 根据在主节点上执行show master status;得出的gtid。</span><br><span class="line">mysql> set global gtid_purged = '95744fa5-ac95-11ea-a21d-0242ac110003:1-2,ce9be252-2b71-11e6-b8f4-00212844f856:1-2';</span><br><span class="line"></span><br><span class="line">mysql> START GROUP_REPLICATION;</span><br></pre></td></tr></table></figure></p><p>查看状态时，从节点一直处于<code>RECOVERING</code>状态<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">mysql> SELECT * FROM performance_schema.replication_group_members;</span><br><span class="line">+---------------------------+--------------------------------------+--------------+-------------+--------------+-------------+----------------+</span><br><span class="line">| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |</span><br><span class="line">+---------------------------+--------------------------------------+--------------+-------------+--------------+-------------+----------------+</span><br><span class="line">| group_replication_applier | 95744fa5-ac95-11ea-a21d-0242ac110003 | 4af6073b5ee9 | 3306 | ONLINE | PRIMARY | 8.0.20 |</span><br><span class="line">| group_replication_applier | 9bda89f2-ac95-11ea-9c6a-0242ac110004 | 762f16c6d69f | 3306 | RECOVERING | SECONDARY | 8.0.20 |</span><br><span class="line">| group_replication_applier | 9f569a5e-ac95-11ea-9ffc-0242ac110005 | 927681f0afc0 | 3306 | RECOVERING | SECONDARY | 8.0.20 |</span><br><span class="line">+---------------------------+--------------------------------------+--------------+-------------+--------------+-------------+----------------+</span><br></pre></td></tr></table></figure></p><p>这是由于找不到从节点导致。<br>1、首先检查一下<code>/etc/hosts</code>文件，看是否将所有容器信息都给补充了。<br>2、执行以下命令重启一下从节点。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">mysql> stop group_replication;</span><br><span class="line">mysql> set global group_replication_recovery_get_public_key=ON;</span><br><span class="line">mysql> start group_replication;</span><br></pre></td></tr></table></figure></p>]]></content>
<summary type="html">
<p>最近在学习MySQL 8.0 新特性，以及MGR。于是想在搭建一套环境来测试。<br>本来在买的阿里云服务器上进行搭建，结果好像是一台机器搭建不了，至少需要三台，于是就放弃了。<br>正好小伙伴分享了docker，于是就利用docker在本地搭建了一套8.0的MGR环境。</p>
<h2 id="准备docker镜像"><a href="#准备docker镜像" class="headerlink" title="准备docker镜像"></a>准备docker镜像</h2><p>运行一个test名称的centos系统镜像<br><code>➜ ~ docker run -d -it --name test centos</code><br>进入容器<br><code>➜ ~ docker exec -it test bash</code><br>在容器内安装MySQL<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[root@c610b533f3dc /]# yum install yum-utils wget -y</span><br><span class="line">[root@c610b533f3dc /]# wget https://dev.mysql.com/get/mysql80-community-release-el7-1.noarch.rpm</span><br><span class="line">[root@c610b533f3dc /]# rpm -ivh mysql80-community-release-el7-1.noarch.rpm</span><br><span class="line">[root@c610b533f3dc /]# yum install -y mysql-server</span><br></pre></td></tr></table></figure></p>
<p>安装完后退出容器，并生成镜像<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@c610b533f3dc /]# exit</span><br><span class="line">➜ ~ docker commit 容器ID mysql8.0.20</span><br></pre></td></tr></table></figure></p>
<h2 id="启动MySQL容器"><a href="#启动MySQL容器" class="headerlink" title="启动MySQL容器"></a>启动MySQL容器</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">docker run -d -it --privileged --name=mysql-mgr-node1 mysql8.0.20 /usr/sbin/init</span><br><span class="line">docker run -d -it --privileged --name=mysql-mgr-node2 mysql8.0.20 /usr/sbin/init</span><br><span class="line">docker run -d -it --privileged --name=mysql-mgr-node3 mysql8.0.20 /usr/sbin/init</span><br></pre></td></tr></table></figure>
</summary>
<category term="MySQL" scheme="http://yoursite.com/categories/MySQL/"/>
<category term="MySQL8.0" scheme="http://yoursite.com/tags/MySQL8-0/"/>
<category term="Docker" scheme="http://yoursite.com/tags/Docker/"/>
</entry>
<entry>
<title>MySQL机制之Double Write</title>
<link href="http://yoursite.com/2020/06/07/new/MySQL/MySQL%E6%9C%BA%E5%88%B6%E4%B9%8BDouble-Write/"/>
<id>http://yoursite.com/2020/06/07/new/MySQL/MySQL机制之Double-Write/</id>
<published>2020-06-07T09:11:55.000Z</published>
<updated>2020-06-11T09:37:51.068Z</updated>
<content type="html"><![CDATA[<h2 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h2><p>我们知道MySQL在更新数据的时候，会先写入到内存中的数据页，然后再在一定的时机时，将脏页给刷新到磁盘中。<br>一个数据页默认为16k，但是由于操作系统的限制，每次需要多次才能将数据页给刷新到磁盘；如果在这个过程中发生故障，导致一个数据页并没有完整的给刷新到磁盘中。我们就称为<code>页断裂</code> 。在这种情况下，我们并不知道会出现什么未知的现象。<br>为了在页断裂场景发生时，能够对损坏的数据页进行修复工作。就引入了<code>两次写（Double Write）</code>机制。</p><h2 id="什么是两次写"><a href="#什么是两次写" class="headerlink" title="什么是两次写"></a>什么是两次写</h2><p>简单来说，就是在对数据页刷盘操作之前，先将该数据页写到一块独立的物理文件位置(ibdata)中。然后再将数据页进行刷盘操作。这样在宕机时，如果出现物理文件损坏，就能够利用ibdata中的副本文件进行修复行为。</p><p>Double Write由两部分组成；一部分是内存中的double write buffer，其大小为2M。另一部分由共享表空间(ibdata)中连续的128页，即2个区组成，大小也是2M。<br>其刷盘流程为：</p><ol><li>当多个数据页需要进行刷盘时，并不直接写入到磁盘的物理文件中，而是先拷贝到内存中的double write buffer中。</li><li>接着从double write buffer中分两次写入到磁盘的共享表空间中(连续存储，顺序写，性能很高)，每次写1M。</li><li>等第二步完成后，再将double write buffer中的脏页数据写入到实际的各个表空间中(离散写)</li></ol><p><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gfio3072ivj30j60dmabz.jpg" alt=""></p><a id="more"></a><h2 id="如何修复"><a href="#如何修复" class="headerlink" title="如何修复"></a>如何修复</h2><p>在发生故障时，如何进行修复操作？<br>1、如果是写doublewrite buffer本身失败,那么这些数据不会被写到磁盘,InnoDB此时会从磁盘载入原始的数据,然后通过InnoDB的事务日志来计算出正确的数据,重新 写入到doublewrite buffer。<br>2、如果 doublewrite buffer写成功的话,但是写磁盘失败,InnoDB就不用通过事务日志来计算了,而是直接用double write buffer的数据再写一遍。<br>3、在恢复的时候,InnoDB直接比较页面的checksum,如果不对的话,就从硬盘载入原始数据,再由事务日志 开始推演出正确的数据.所以InnoDB的恢复通常需要较长的时间.<br><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gfio31sa8aj30il0bhjsr.jpg" alt=""></p><h2 id="5-7优化"><a href="#5-7优化" class="headerlink" title="5.7优化"></a>5.7优化</h2><p>在《MySQL运维内参》一书中，提到了在5.7中，对double write进行了优化<code>批量刷盘</code>。但是我在5.7.18版本中，并没有看到有那个参数<code>innodb_doublewrite_batch_size</code>（为MySQL 8.0.20引入）。</p><p><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gfjtvjy3c0j30fa09k75f.jpg" alt=""></p><p>批量刷盘包括两种方式，分别是LRU(Least Recently Used,最近最少使用)方式和LIST方式。当Buffer Pool空间不足时，再载入新的页面就必须要将一些不怎么用到的、旧的页面淘汰出去，此时系统就会从LRU链表中找到最老的页面，进行批量刷盘，将释放的空间加入到空闲空间中去，这种情况就是LRU刷盘。当日志空间不足，或者是后台MASTER线程在定时刷盘时，不需要区分页面的新旧状态，只需要选择LSN最小的那些页面，从前到后刷一批页面到文件中，此时所用的策略就是LIST方式。 </p><p>在批量刷盘的两次写中，这两种刷盘方法对应的两次写空间互不干涉。 </p><p>从图中可以看出落到最终的每一个shard,其实就是一个batch，对应的参数就是innodb_doublewrite_batch_size。一个shard，有一个数组，长度为innodb_doublewrite_batch_size，与单一页面刷盘的两次写是一样的，只是这个数组只属于一个shard而已。 </p><p>假设由于页面淘汰，系统要做一次批量刷盘，这次就是LRU方式的，那么此时系统就需要将当前页面加入到两次写缓存中，首先根据当前页面所在的Instance号及刷盘类型就可以找到对应的shard缓存，找到缓存后，判断当前shard是否已经满了，即是否已经达到innodb_doublewrite_batch_size的大小，如果没有达到，则将当前页面内容追加复制到当前的shard缓存中，这样当前页面的刷盘操作就完成了。这里并不像单一页面那样，先写入缓存空间中，然后写入ibdata文件的两次写空间，最后还需要立即将页面的真实内容刷入表空间，对于批量刷盘来说，只需要写入到shard缓存即可。 </p><p>如果当前shard中缓存的页面个数已经达到了innodb_doublewrite_batch_size，则说明当前缓存空间已经满了，此时不得不将当前shard缓存的页面写入两次写文件中，写完之后再将两次写文件FLUSH到磁盘，最后将对应的真实页面刷盘，此时可能是随机写入了，因为对应的两次写缓存中虽然是连续的，但对应的真实页面就不会这样了。这里需要注意的一点就是，表空间页面的刷盘，是异步IO操作，此时需要等待异步IO完成，且整个shard中的页面都刷盘后，刷盘操作才可以继续向后执行，而这个shard也可以再次重新使用了，缓存中的数据也都会被清空。</p><p>需要注意的是，上面过程中写入是连续innodb_doublewrite_batch_size 个页面，所以性能会比写入多次而每次写入一个页面的情况好很多。批量刷盘的情况下，有可能每隔innodb_doublewrite_batch_size个页面的刷盘操作，就会出现一次等待操作，且等待时间长短不一定，但这也是在单一页面刷盘的基础上优化过的，做了改进。</p><h2 id="思考"><a href="#思考" class="headerlink" title="思考"></a>思考</h2><h3 id="发生页断裂时，理论上通过redo就应该能够进行恢复了，为什么还需要double-write？"><a href="#发生页断裂时，理论上通过redo就应该能够进行恢复了，为什么还需要double-write？" class="headerlink" title="发生页断裂时，理论上通过redo就应该能够进行恢复了，为什么还需要double write？"></a>发生页断裂时，理论上通过redo就应该能够进行恢复了，为什么还需要double write？</h3><p>这个就得从redo log保存的记录格式说起了。<br>redo日志主要采用的是物理日志和逻辑日志两种方式。<br>对于能够唯一确定数据存储在磁盘位置时，使用的是物理日志，通过(group_id,file_id,page_no,offset)4元组来确定位置，并记录数据页变更。在这种记录格式下，能够通过redo log进行问题修复。<br>但是，对于一个操作记录产生的日志跨越了多个数据页时，那么会产生多个物理页面的日志，但对于每个物理页面日志，里面记录则是逻辑信息。在这种情况下，就无法单靠redo log进行问题修复</p><h3 id="性能损耗"><a href="#性能损耗" class="headerlink" title="性能损耗"></a>性能损耗</h3><p>如果开启了double write机制。那么每次刷盘都会多一次内存复制和持久化操作。那么会对性能造成多大的影响呢？<br>由于持久化刷盘过程是顺序写，性能很高。据官方介绍，可能会有10%的性能损耗，但是为了数据得到完整性，这点损耗还是很有必要的。</p><blockquote><p><a href="https://www.cnblogs.com/xuliuzai/p/10290196.html" target="_blank" rel="noopener">https://www.cnblogs.com/xuliuzai/p/10290196.html</a><br><a href="https://mp.weixin.qq.com/s/9GHIpT_YeUoZNJ2X6cS-YQ" target="_blank" rel="noopener">https://mp.weixin.qq.com/s/9GHIpT_YeUoZNJ2X6cS-YQ</a> （推荐）</p></blockquote>]]></content>
<summary type="html">
<h2 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h2><p>我们知道MySQL在更新数据的时候，会先写入到内存中的数据页，然后再在一定的时机时，将脏页给刷新到磁盘中。<br>一个数据页默认为16k，但是由于操作系统的限制，每次需要多次才能将数据页给刷新到磁盘；如果在这个过程中发生故障，导致一个数据页并没有完整的给刷新到磁盘中。我们就称为<code>页断裂</code> 。在这种情况下，我们并不知道会出现什么未知的现象。<br>为了在页断裂场景发生时，能够对损坏的数据页进行修复工作。就引入了<code>两次写（Double Write）</code>机制。</p>
<h2 id="什么是两次写"><a href="#什么是两次写" class="headerlink" title="什么是两次写"></a>什么是两次写</h2><p>简单来说，就是在对数据页刷盘操作之前，先将该数据页写到一块独立的物理文件位置(ibdata)中。然后再将数据页进行刷盘操作。这样在宕机时，如果出现物理文件损坏，就能够利用ibdata中的副本文件进行修复行为。</p>
<p>Double Write由两部分组成；一部分是内存中的double write buffer，其大小为2M。另一部分由共享表空间(ibdata)中连续的128页，即2个区组成，大小也是2M。<br>其刷盘流程为：</p>
<ol>
<li>当多个数据页需要进行刷盘时，并不直接写入到磁盘的物理文件中，而是先拷贝到内存中的double write buffer中。</li>
<li>接着从double write buffer中分两次写入到磁盘的共享表空间中(连续存储，顺序写，性能很高)，每次写1M。</li>
<li>等第二步完成后，再将double write buffer中的脏页数据写入到实际的各个表空间中(离散写)</li>
</ol>
<p><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gfio3072ivj30j60dmabz.jpg" alt=""></p>
</summary>
<category term="MySQL" scheme="http://yoursite.com/categories/MySQL/"/>
<category term="MySQL机制" scheme="http://yoursite.com/tags/MySQL%E6%9C%BA%E5%88%B6/"/>
</entry>
<entry>
<title>pt-osc死锁分析记录两则</title>
<link href="http://yoursite.com/2020/06/04/new/MySQL/pt-osc%E6%AD%BB%E9%94%81%E5%88%86%E6%9E%90%E8%AE%B0%E5%BD%95%E4%B8%A4%E5%88%99/"/>
<id>http://yoursite.com/2020/06/04/new/MySQL/pt-osc死锁分析记录两则/</id>
<published>2020-06-04T15:56:06.000Z</published>
<updated>2020-06-04T15:59:10.595Z</updated>
<content type="html"><![CDATA[<h2 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h2><p>今天看到叶老师公众号推送文章《<a href="!https://mp.weixin.qq.com/s/mJ6a-sV2C2ru5pA5QNAAZQ">pt-osc在线重建表导致死锁的分析及对应的优化方案</a>》，让我想起了前段时间同样遇到了pt-osc改表导致的死锁。故此记录一下。</p><h2 id="pt-osc"><a href="#pt-osc" class="headerlink" title="pt-osc"></a>pt-osc</h2><p>pt-online-schema-change：PERCONA提供的在线改表工具，避免MySQL在执行改表操作时造成的锁表和主从延迟情况发生。</p><h3 id="工作原理"><a href="#工作原理" class="headerlink" title="工作原理"></a>工作原理</h3><p>1、创建一个跟原表表结构一样的新表。取名为<code>_oldTableNmae_new</code><br>2、修改新表结构<br>3、在原表中创建insert、update、delete三个类型的触发器，用于做增量数据迁移。原表SQL和触发器触发SQL在同一事务当中。<br>4、以一定块大小(chunk-size)从原表拷贝数据到新表<br>5、数据拷贝完后，修改表名：<code>rename table to table_old; rename _table_new to table</code><br>6、删除old表，删除三个触发器</p><h3 id="版本变化"><a href="#版本变化" class="headerlink" title="版本变化"></a>版本变化</h3><p>3.0.2之前的update触发器：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">REPLACE INTO `lc`.`_hb_new` (`id`, `ts`, `ts2`, `c1`) VALUES (NEW.`id`, NEW.`ts`, NEW.`ts2`, NEW.`c1`)</span><br></pre></td></tr></table></figure></p><p>3.0.2之后的update触发器:<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">BEGIN</span><br><span class="line"> DELETE IGNORE FROM `lc`.`_hb_new` WHERE !(OLD.`id` <=> NEW.`id`) AND `lc`.`_hb_new`.`id` <=> OLD.`id`;</span><br><span class="line"> REPLACE INTO `lc`.`_hb_new` (`id`, `ts`, `ts2`) VALUES (NEW.`id`, NEW.`ts`, NEW.`ts2`);</span><br><span class="line">END</span><br></pre></td></tr></table></figure></p><a id="more"></a><h2 id="死锁案例一"><a href="#死锁案例一" class="headerlink" title="死锁案例一"></a>死锁案例一</h2><p>pt-osc改表操作与业务insert语句形成死锁，业务insert操作回滚。</p><h3 id="环境"><a href="#环境" class="headerlink" title="环境"></a>环境</h3><p>MySQL5.7.26、RC隔离级别、innodb_autoinc_lock_mode=1</p><p>表结构：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">CREATE TABLE `t` (</span><br><span class="line"> `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '自增主键',</span><br><span class="line"> `c1` int(11) NOT NULL DEFAULT '0' COMMENT 'c1',</span><br><span class="line"> `c2` int(11) NOT NULL DEFAULT '0' COMMENT 'c2'</span><br><span class="line"> `c3` int(11) NOT NULL DEFAULT '0' COMMENT 'c3'</span><br><span class="line"> PRIMARY KEY (`id`)</span><br><span class="line">) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='';</span><br></pre></td></tr></table></figure></p><h3 id="死锁日志"><a href="#死锁日志" class="headerlink" title="死锁日志"></a>死锁日志</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><span class="line">2020-04-26T06:24:05.340343+08:00 733947 [Note] InnoDB: Transactions deadlock detected, dumping detailed information.</span><br><span class="line">2020-04-26T06:24:05.341246+08:00 733947 [Note] InnoDB: </span><br><span class="line">*** (1) TRANSACTION:</span><br><span class="line"></span><br><span class="line">TRANSACTION 918773485, ACTIVE 0 sec setting auto-inc lock</span><br><span class="line">mysql tables in use 2, locked 2</span><br><span class="line">LOCK WAIT 4 lock struct(s), heap size 1136, 1 row lock(s), undo log entries 2</span><br><span class="line">MySQL thread id 668554, OS thread handle 139777592952576, query id 2675769996 192.168.1.1 test_user update</span><br><span class="line">REPLACE INTO `test_db`.`_t_new` (`id`, `c1`, `c2`, `c3`) VALUES (NEW.`id`, NEW.`c1`, NEW.`c2`, NEW.`c3`)</span><br><span class="line">2020-04-26T06:24:05.341319+08:00 733947 [Note] InnoDB: *** (1) WAITING FOR THIS LOCK TO BE GRANTED:</span><br><span class="line"></span><br><span class="line">TABLE LOCK table `test_db`.`_t_new` trx id 918773485 lock mode AUTO-INC waiting</span><br><span class="line"></span><br><span class="line">2020-04-26T06:24:05.341351+08:00 733947 [Note] InnoDB: *** (2) TRANSACTION:</span><br><span class="line"></span><br><span class="line">TRANSACTION 918773482, ACTIVE 1 sec fetching rows</span><br><span class="line">mysql tables in use 2, locked 2</span><br><span class="line">120 lock struct(s), heap size 24784, 8894 row lock(s), undo log entries 8099</span><br><span class="line">MySQL thread id 733947, OS thread handle 139777597007616, query id 2675769985 localhost root Sending data</span><br><span class="line">INSERT LOW_PRIORITY IGNORE INTO `test_db`.`_t_new` (`id`, `c1`, `c2`, `c3`) SELECT `id`, `c1`, `c2`, `c3` FROM </span><br><span class="line">`test_db`.`t` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '95439963')) AND ((`id` <= '95448404')) LOCK IN SHARE MODE </span><br><span class="line">2020-04-26T06:24:05.341398+08:00 733947 [Note] InnoDB: *** (2) HOLDS THE LOCK(S):</span><br><span class="line"></span><br><span class="line">TABLE LOCK table `test_db`.`_t_new` trx id 918773482 lock mode AUTO-INC</span><br><span class="line">2020-04-26T06:24:05.341415+08:00 733947 [Note] InnoDB: *** (2) WAITING FOR THIS LOCK TO BE GRANTED:</span><br><span class="line"></span><br><span class="line">RECORD LOCKS space id 974 page no 145414 n bits 80 index PRIMARY of table `test_db`.`t` trx id 918773482 lock mode S locks rec but not gap waiting</span><br><span class="line">Record lock, heap no 9 PHYSICAL RECORD: n_fields 27; compact format; info bits 0</span><br><span class="line">0: len 4; hex 85b06d55; asc mU;; --'5b06d55'从16进制转换为10进制,得到的值为 95448405</span><br><span class="line"></span><br><span class="line">1: len 4; hex 80002712; asc ;;</span><br><span class="line">2: len 4; hex 800c24d7; asc $ ;;</span><br><span class="line">3: len 4; hex 80000003; asc ;;</span><br><span class="line"></span><br><span class="line">2020-04-26T06:24:05.342491+08:00 733947 [Note] InnoDB: *** WE ROLL BACK TRANSACTION (1)</span><br></pre></td></tr></table></figure><h3 id="死锁分析"><a href="#死锁分析" class="headerlink" title="死锁分析"></a>死锁分析</h3><p>事务：918773482<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">当前的SQL语句: </span><br><span class="line"> INSERT LOW_PRIORITY IGNORE INTO `test_db`.`_t_new` (`id`, `c1`, `c2`, `c3`) SELECT `id`, `c1`, `c2`, `c3` FROM </span><br><span class="line"> `test_db`.`t` FORCE INDEX(`PRIMARY`) WHERE ((`id` >= '95439963')) AND ((`id` <= '95448404')) LOCK IN SHARE MODE; </span><br><span class="line"></span><br><span class="line">持有的锁信息：</span><br><span class="line"> TABLE LOCK table `test_db`.`_t_new` ... lock mode AUTO-INC --表示持有表_t_new上的自增长锁;</span><br><span class="line"></span><br><span class="line">在等待的锁信息:</span><br><span class="line"> index PRIMARY of table `test_db`.`t` --表示在等的是表t的主键索引上面的锁;</span><br><span class="line"> lock_mode S locks rec but not gap waiting --表示需要加一个共享锁(读锁)，当前的状态是等待中;</span><br><span class="line"> 0: len 4; hex 85b06d55; asc mU;; --主键字段, 5b06d55的16进制为转换为10进制得到值为: 95448405; </span><br><span class="line"></span><br><span class="line">通过分析得知：</span><br><span class="line"> TRANSACTION 918773482持有的锁: 表_t_new的自增长锁; </span><br><span class="line"> TRANSACTION 918773482在等待TRANSACTION 918773485的锁: 表t的主键索引primary: record lock: id=95448405。</span><br></pre></td></tr></table></figure></p><p>事务：918773485<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">当前的SQL语句: </span><br><span class="line"> REPLACE INTO `test_db`.`_t_new` (`id`, `c1`, `c2`, `c3`) VALUES (NEW.`id`, NEW.`c1`, NEW.`c2`, NEW.`c3`);</span><br><span class="line"></span><br><span class="line">持有的锁信息:</span><br><span class="line"> 根据TRANSACTION 918773482在等待TRANSACTION 918773485的锁为 primary: record lock: id=95448405 的行锁，所以推导出TRANSACTION 918773485持有表t主键索引 id=95448405 的行锁;</span><br><span class="line"></span><br><span class="line">在等待的锁信息:</span><br><span class="line"> TABLE LOCK table `test_db`.`_t_new` ... lock mode AUTO-INC waiting --表示在等表_t_new上的自增长锁;</span><br><span class="line"></span><br><span class="line">通过分析得知： </span><br><span class="line"> TRANSACTION 918773485持有的锁：持有表t主键索引primary id=95448405 的行锁；</span><br><span class="line"> TRANSACTION 918773485在等待TRANSACTION 918773482的锁: 表_t_new上的自增长锁。</span><br></pre></td></tr></table></figure></p><table><thead><tr><th style="text-align:center">时间点</th><th style="text-align:center">918773482<br>pt-osc改表批量导入语句</th><th style="text-align:center">918773485918773485<br>业务正常插入操作</th></tr></thead><tbody><tr><td style="text-align:center"></td><td style="text-align:center">begin</td><td style="text-align:center">begin</td></tr><tr><td style="text-align:center"></td><td style="text-align:center">INSERT LOW_PRIORITY IGNORE INTO _t_new (id, c1, c2, c3) SELECT id, c1, c2, c3 FROM test_db.t FORCE INDEX(PRIMARY) WHERE ((id >= ‘95439963’)) AND ((id <= ‘95448404’)) LOCK IN SHARE MODE;</td><td style="text-align:center"></td></tr><tr><td style="text-align:center">T1</td><td style="text-align:center">持有的锁: 表_t_new: AUTO-INC</td><td style="text-align:center"></td></tr><tr><td style="text-align:center">T2</td><td style="text-align:center"></td><td style="text-align:center">INSERT INTO t (c1, c2, c3) VALUES (0, 0, 0);<br>持有表t主键索引 id=95448405 的行锁(排他X锁)</td></tr><tr><td style="text-align:center">T3</td><td style="text-align:center"></td><td style="text-align:center">REPLACE INTO _t_new (id, c1, c2, c3) VALUES (NEW.id, NEW.c1, NEW.c2, NEW.c3);<br>等待的锁: 表_t_new: AUTO-INC</td></tr><tr><td style="text-align:center">T4</td><td style="text-align:center">等待的锁: 表t: primary: record lock: id=95448405</td></tr></tbody></table><p>T3被T1阻塞，T4被T2阻塞，因此锁资源请求形成了环路，进而触发死锁检测，MySQL会把执行代价最小的事务回滚掉，让其它事务得以继续进行;</p><h3 id="思考"><a href="#思考" class="headerlink" title="思考"></a>思考</h3><ol><li>为什么pt-osc会去申请原表t的主键锁呢？<br>因为 id <= 95448404 是范围等值查询并且id=95448404是当前主键索引的最大值 ， 锁的过程实际上是id<=95448404的下一条记录也就是这个索引页的最大记录supremum（如果这个在RC隔离级别下没有被锁，则会立即释放），需要访问到 id=95448405 才会停止下来，所以需要申请 持有 id=95448405 的行锁 ，因此被 T2时刻 的SQL语句阻塞。</li></ol><p>TRANSACTION 918773482的事务语句是pt-osc拷贝的最后一个chunk-size，并且期间其它事务有对原表做insert操作， 所以才会发生死锁。</p><h2 id="死锁案例二"><a href="#死锁案例二" class="headerlink" title="死锁案例二"></a>死锁案例二</h2><p>pt-osc改表操作与业务update语句形成死锁，业务update操作回滚。</p><h3 id="环境-1"><a href="#环境-1" class="headerlink" title="环境"></a>环境</h3><p>阿里云MySQL5.6、RC隔离级别</p><p>表结构：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">CREATE TABLE `t` (</span><br><span class="line"> `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '自增主键',</span><br><span class="line"> `c1` int(11) NOT NULL DEFAULT '0' COMMENT 'c1',</span><br><span class="line"> `c2` int(11) NOT NULL DEFAULT '0' COMMENT 'c2'</span><br><span class="line"> `c3` int(11) NOT NULL DEFAULT '0' COMMENT 'c3'</span><br><span class="line"> PRIMARY KEY (`id`),</span><br><span class="line"> UNIQUE KEY `idx_c1_c2(`c1`,`c2`)</span><br><span class="line">) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='';</span><br></pre></td></tr></table></figure></p><h3 id="死锁日志-1"><a href="#死锁日志-1" class="headerlink" title="死锁日志"></a>死锁日志</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line">2020-05-11 15:04:22 7f728ebff700</span><br><span class="line">*** (1) TRANSACTION:</span><br><span class="line">TRANSACTION 582527418, ACTIVE 0.002 sec setting auto-inc lock</span><br><span class="line">mysql tables in use 2, locked 2</span><br><span class="line">LOCK WAIT 4 lock struct(s), heap size 1184, 1 row lock(s), undo log entries 2</span><br><span class="line">LOCK BLOCKING MySQL thread id: 34439719 block 17762714</span><br><span class="line">MySQL thread id 17762714, OS thread handle 0x7f728eaba700, query id 323487893 192.168.55.8 test update</span><br><span class="line">REPLACE INTO `test`.`_t_new` (`id`, `c1`, `c2`, `c3`) VALUES (NEW.`id`, NEW.`c1`, NEW.`c2`, NEW.`c3`, NEW.`c4`)</span><br><span class="line">*** (1) WAITING FOR THIS LOCK TO BE GRANTED:</span><br><span class="line">TABLE LOCK table `test`.`_t_new` trx id 582527418 lock mode AUTO-INC waiting</span><br><span class="line">*** (2) TRANSACTION:</span><br><span class="line">TRANSACTION 582527416, ACTIVE 0.012 sec inserting</span><br><span class="line">mysql tables in use 2, locked 2</span><br><span class="line">9 lock struct(s), heap size 1184, 6 row lock(s), undo log entries 3</span><br><span class="line">MySQL thread id 34439719, OS thread handle 0x7f728ebff700, query id 323487888 192.168.58.10 test update</span><br><span class="line">REPLACE INTO `test`.`_t_new` (`id`, `c1`, `c2`, `c3`) VALUES (NEW.`id`, NEW.`c1`, NEW.`c2`, NEW.`c3`)</span><br><span class="line">*** (2) HOLDS THE LOCK(S):</span><br><span class="line">TABLE LOCK table `test`.`_t_new` trx id 582527416 lock mode AUTO-INC</span><br><span class="line">*** (2) WAITING FOR THIS LOCK TO BE GRANTED:</span><br><span class="line">RECORD LOCKS space id 51 page no 467072 n bits 224 index `idx_c1_c2` of table `test`.`_t_new` trx id 582527416 lock_mode X waiting</span><br><span class="line">Record lock, heap no 154 PHYSICAL RECORD: n_fields 4; compact format; info bits 0</span><br><span class="line"> 0: len 8; hex 000000000f97b07e; asc ~;;</span><br><span class="line"> 1: len 1; hex 08; asc ;;</span><br><span class="line"> 2: len 1; hex 00; asc ;;</span><br><span class="line"> 3: len 8; hex 0000000005f1acf1; asc ;;</span><br><span class="line"></span><br><span class="line">*** WE ROLL BACK TRANSACTION (1)</span><br></pre></td></tr></table></figure><h3 id="死锁分析-1"><a href="#死锁分析-1" class="headerlink" title="死锁分析"></a>死锁分析</h3><p>事务：582527416<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">原业务语句：</span><br><span class="line">UPDATE t SET c3 = 'xx' WHERE c1 = 'xx' AND c2 = 'xx' LIMIT 1</span><br><span class="line"></span><br><span class="line">经过触发器改写后语句：</span><br><span class="line">begin</span><br><span class="line">UPDATE t SET c3 = 'xx' WHERE c1 = 'xx' AND c2 = 'xx' LIMIT 1</span><br><span class="line">DELETE IGNORE FROM `test`.`_t_new` WHERE !(OLD.`id` <=> NEW.`id`) AND `test`.`_t_new`.`id` <=> OLD.`id`</span><br><span class="line">REPLACE INTO `test`.`_t_new` (`id`, `c1`, `c2`, `c3`) VALUES (NEW.`id`, NEW.`c1`, NEW.`c2`, NEW.`c3`); </span><br><span class="line">end</span><br><span class="line"></span><br><span class="line">锁分析：</span><br><span class="line">当前持有 lock mode AUTO-INC 锁（表级别）</span><br><span class="line">由于存在唯一索引，所以在插入的时候会进行唯一性检测，此时需要获取 next-key lock</span><br></pre></td></tr></table></figure></p><p>事务：TRANSACTION 582527418<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">原业务语句：</span><br><span class="line">UPDATE t SET c3 = 'xx' WHERE c1 = 'xx' AND c2 = 'xx' LIMIT 1</span><br><span class="line"></span><br><span class="line">经过触发器改写后语句：</span><br><span class="line">begin</span><br><span class="line">UPDATE t SET c3 = 'xx' WHERE c1 = 'xx' AND c2 = 'xx' LIMIT 1</span><br><span class="line">DELETE IGNORE FROM `test`.`_t_new` WHERE !(OLD.`id` <=> NEW.`id`) AND `test`.`_t_new`.`id` <=> OLD.`id`</span><br><span class="line">REPLACE INTO `test`.`_t_new` (`id`, `c1`, `c2`, `c3`) VALUES (NEW.`id`, NEW.`c1`, NEW.`c2`, NEW.`c3`); </span><br><span class="line">end</span><br><span class="line"></span><br><span class="line">锁分析：</span><br><span class="line">持有记录的x锁和插入意向锁 等待_t_new表级别的auto-inc 锁</span><br></pre></td></tr></table></figure></p><table><thead><tr><th style="text-align:center">时间点</th><th style="text-align:center">sesson 1<br>582527416</th><th style="text-align:center">session 2<br>582527418</th></tr></thead><tbody><tr><td style="text-align:center"></td><td style="text-align:center">begin</td><td style="text-align:center">begin</td></tr><tr><td style="text-align:center">T1</td><td style="text-align:center">REPLACE INTO <code>test</code>.<code>_t_new</code> (<code>id</code>, <code>c1</code>, <code>c2</code>, <code>c3</code>) VALUES (NEW.<code>id</code>, NEW.<code>c1</code>, NEW.<code>c2</code>, NEW.<code>c3</code>); <br>插入前检查，唯一键上的插入意向锁</td><td style="text-align:center"></td></tr><tr><td style="text-align:center">T2</td><td style="text-align:center"></td><td style="text-align:center">REPLACE INTO <code>test</code>.<code>_t_new</code> (<code>id</code>, <code>c1</code>, <code>c2</code>, <code>c3</code>) VALUES (NEW.<code>id</code>, NEW.<code>c1</code>, NEW.<code>c2</code>, NEW.<code>c3</code>);<br>插入前检查，唯一键上的插入意向锁</td></tr><tr><td style="text-align:center">T3</td><td style="text-align:center">执行插入，拥有_t_new的auto-inc锁，等待唯一键插入意向锁释放</td><td style="text-align:center"></td></tr><tr><td style="text-align:center">T4</td><td style="text-align:center"></td><td style="text-align:center">执行插入，等待_t_new的auto-inc锁</td></tr></tbody></table><h3 id="思考-1"><a href="#思考-1" class="headerlink" title="思考"></a>思考</h3><ol><li>为什么平时业务同样的update，但是只有在改表时才会触发死锁呢？<br>因为改表过程中，是增量将数据导入，此时还未导入到update所操作的id上。业务update执行时，触发器转换后的语句在执行时，会申请_t_new表的最大值之间的auto-inc锁。当业务并发大时可能就会造成业务update语句之间的死锁情况。</li></ol><h2 id="如何避免"><a href="#如何避免" class="headerlink" title="如何避免"></a>如何避免</h2><ol><li>设置pt-osc的chunk-size为更小的值，可以减少死锁的发生，但是不可能避免死锁的发生。</li><li>如果参数innodb_autoinc_lock_mode的值为2，大大降低死锁发生的概率，原因如下：<br>造成本案例死锁的原因之一就是在参数innodb_autoinc_lock_mode=1的环境下，持有的自增锁直到SQL语句结束后才释放；<br>如果参数innodb_autoinc_lock_mode=2，自增锁在申请后就释放，不需要等语句结束，大大缩短了持有自增锁的时间，从而降低了死锁发生的概率。</li><li>数据库版本为MySQL 8.0.18或者以上， 事务隔离为RR可重复读则不会出现本案例的死锁，原因如下：<br>8.0.18或者以上的版本中，对加锁规则有一个优化：在RR可重复读级别下，唯一索引上的范围查询，不再需要访问到不满足条件的第一个值为止（即不再需要对不必要的数据上锁）。在叶老师的这篇文章中有说明：<a href="https://mp.weixin.qq.com/s/xDKKuIvVgFNiKp5kt2NIgA" target="_blank" rel="noopener">https://mp.weixin.qq.com/s/xDKKuIvVgFNiKp5kt2NIgA</a> InnoDB这个将近20年的”bug”修复了；</li></ol><blockquote><p><a href="https://mp.weixin.qq.com/s/mJ6a-sV2C2ru5pA5QNAAZQ" target="_blank" rel="noopener">https://mp.weixin.qq.com/s/mJ6a-sV2C2ru5pA5QNAAZQ</a><br><a href="https://dev.mysql.com/doc/refman/8.0/en/innodb-locks-set.html" target="_blank" rel="noopener">https://dev.mysql.com/doc/refman/8.0/en/innodb-locks-set.html</a></p></blockquote>]]></content>
<summary type="html">
<h2 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h2><p>今天看到叶老师公众号推送文章《<a href="!https://mp.weixin.qq.com/s/mJ6a-sV2C2ru5pA5QNAAZQ">pt-osc在线重建表导致死锁的分析及对应的优化方案</a>》，让我想起了前段时间同样遇到了pt-osc改表导致的死锁。故此记录一下。</p>
<h2 id="pt-osc"><a href="#pt-osc" class="headerlink" title="pt-osc"></a>pt-osc</h2><p>pt-online-schema-change：PERCONA提供的在线改表工具，避免MySQL在执行改表操作时造成的锁表和主从延迟情况发生。</p>
<h3 id="工作原理"><a href="#工作原理" class="headerlink" title="工作原理"></a>工作原理</h3><p>1、创建一个跟原表表结构一样的新表。取名为<code>_oldTableNmae_new</code><br>2、修改新表结构<br>3、在原表中创建insert、update、delete三个类型的触发器，用于做增量数据迁移。原表SQL和触发器触发SQL在同一事务当中。<br>4、以一定块大小(chunk-size)从原表拷贝数据到新表<br>5、数据拷贝完后，修改表名：<code>rename table to table_old; rename _table_new to table</code><br>6、删除old表，删除三个触发器</p>
<h3 id="版本变化"><a href="#版本变化" class="headerlink" title="版本变化"></a>版本变化</h3><p>3.0.2之前的update触发器：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">REPLACE INTO `lc`.`_hb_new` (`id`, `ts`, `ts2`, `c1`) VALUES (NEW.`id`, NEW.`ts`, NEW.`ts2`, NEW.`c1`)</span><br></pre></td></tr></table></figure></p>
<p>3.0.2之后的update触发器:<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">BEGIN</span><br><span class="line"> DELETE IGNORE FROM `lc`.`_hb_new` WHERE !(OLD.`id` &lt;=&gt; NEW.`id`) AND `lc`.`_hb_new`.`id` &lt;=&gt; OLD.`id`;</span><br><span class="line"> REPLACE INTO `lc`.`_hb_new` (`id`, `ts`, `ts2`) VALUES (NEW.`id`, NEW.`ts`, NEW.`ts2`);</span><br><span class="line">END</span><br></pre></td></tr></table></figure></p>
</summary>
<category term="MySQL" scheme="http://yoursite.com/categories/MySQL/"/>
<category term="MySQL" scheme="http://yoursite.com/tags/MySQL/"/>
<category term="PT" scheme="http://yoursite.com/tags/PT/"/>
</entry>
<entry>
<title>MySQL机制之index merge</title>
<link href="http://yoursite.com/2020/05/31/new/MySQL/MySQL%E6%9C%BA%E5%88%B6%E4%B9%8Bindex-merge/"/>
<id>http://yoursite.com/2020/05/31/new/MySQL/MySQL机制之index-merge/</id>
<published>2020-05-31T14:50:20.000Z</published>
<updated>2020-05-31T14:54:57.098Z</updated>
<content type="html"><![CDATA[<h2 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h2><p>查询时where后面可能会涉及到多个字段，它们之间进行AND或OR。在MySQL 5.0之前，一个表一次只能使用一个索引，无法同时使用多个索引分别进行条件扫描。从5.1开始引入了<code>index merge</code>技术。<br><code>index merge</code>技术就是<font color="red">对多个索引分别进行条件扫描，然后将他们各自的结果进行合并</font></p><p>该特性主要会体现在以下场景：</p><ol><li>对OR语句求并集</li><li>对AND语句求交集</li><li>对AND和OR组合语句求结果</li></ol><p>在满足index merge条件下的查询计划中会出现<code>type:index_merge</code>。</p><h2 id="AND取交集-union"><a href="#AND取交集-union" class="headerlink" title="AND取交集(union)"></a>AND取交集(union)</h2><p>union就是多个索引条件扫描，对得到的结果进行并集运算，显然是多个条件之间进行的是 OR 运算。<br>案例：<code>SELECT * FROM tmp_index_merge where key1_part1 = 2 and key2_part1 = 4\G</code></p><h2 id="OR取并集-intersect"><a href="#OR取并集-intersect" class="headerlink" title="OR取并集(intersect)"></a>OR取并集(intersect)</h2><p>intersect就是多个索引条件扫描得到的结果进行交集运算。<br>案例：<code>SELECT * FROM tmp_index_merge where key1_part1 = 2 or key2_part1 = 4\G</code></p><h2 id="AND和OR组合取并集-sort-union"><a href="#AND和OR组合取并集-sort-union" class="headerlink" title="AND和OR组合取并集(sort_union)"></a>AND和OR组合取并集(sort_union)</h2><p>多个条件扫描进行 OR 运算，但是不符合 index union merge算法的，此时可能会使用 sort_union算法。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">SELECT * FROM tbl_name WHERE key_col1 < 10 OR key_col2 < 20;</span><br><span class="line">SELECT * FROM tbl_name WHERE (key_col1 > 10 OR key_col2 = 20) AND nonkey_col=30;</span><br></pre></td></tr></table></figure></p><h2 id="注意"><a href="#注意" class="headerlink" title="注意"></a>注意</h2><p>在MySQL5.6.7之前的版本中，存在<font color="red">range优先原则</font>。只要可以使用Range访问方式，那就不会再使用index merge。</p><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>索引合并机制并不是什么新鲜东西，但是仍然存在很多人认为一条查询只能使用一个索引的观念。本篇文章只是简单的介绍MySQL存在这么一个机制，并未过深的研究。<br>简言之，索引合并会同时利用多个索引进行查询，尽可能得过滤掉不需要的数据行，然后再进行一次统一的回表行为。较少无谓的回表行为。</p><blockquote><p><a href="http://dev.mysql.com/doc/refman/5.6/en/index-merge-optimization.html" target="_blank" rel="noopener">http://dev.mysql.com/doc/refman/5.6/en/index-merge-optimization.html</a><br><a href="https://www.orczhou.com/index.php/2013/01/mysql-source-code-query-optimization-index-merge/" target="_blank" rel="noopener">https://www.orczhou.com/index.php/2013/01/mysql-source-code-query-optimization-index-merge/</a><br><a href="http://www.cnblogs.com/nocode/archive/2013/01/28/2880654.html" target="_blank" rel="noopener">http://www.cnblogs.com/nocode/archive/2013/01/28/2880654.html</a></p></blockquote>]]></content>
<summary type="html">
<h2 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h2><p>查询时where后面可能会涉及到多个字段，它们之间进行AND或OR。在MySQL 5.0之前，一个表一次只能使用一个索引，无法同时使用多个索
</summary>
<category term="MySQL" scheme="http://yoursite.com/categories/MySQL/"/>
<category term="MySQL机制" scheme="http://yoursite.com/tags/MySQL%E6%9C%BA%E5%88%B6/"/>
</entry>
<entry>
<title>MySQL机制介绍之NLJ、BNL、BKA</title>
<link href="http://yoursite.com/2020/05/30/new/MySQL/MySQL%E6%9C%BA%E5%88%B6%E4%BB%8B%E7%BB%8D%E4%B9%8BNLJ%E3%80%81BNL%E3%80%81BKA/"/>
<id>http://yoursite.com/2020/05/30/new/MySQL/MySQL机制介绍之NLJ、BNL、BKA/</id>
<published>2020-05-29T17:03:51.000Z</published>
<updated>2020-05-29T17:05:25.376Z</updated>
<content type="html"><![CDATA[<h2 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h2><p>MySQL 5.5版本前，MySQL本身只支持一种表间关联方式，就是嵌套循环(Nested Loop)。如果关联表的数据量很大，则join关联的执行时间会非常长。<br>在5.5版本中，MySQL通过引入Block Nested-Loop Join(BNL)算法来优化嵌套执行。</p><h2 id="NLJ"><a href="#NLJ" class="headerlink" title="NLJ"></a>NLJ</h2><p>将驱动表的结果集作为循环基础数据，然后循环从该结果集中每次一条获取数据作为下一个表的过滤条件查询数据，然后合并结果。<br>如果有多表join，则将前面的表的结果集作为循环数据，取到每行再到链接的下一个表循环匹配。</p><h3 id="SNLJ"><a href="#SNLJ" class="headerlink" title="SNLJ"></a>SNLJ</h3><p>Simple Nested-Loops Join(SNLJ，简单嵌套循环联接)，该算法比较简单、直接。驱动表中的每一条记录都与被驱动表中的记录进行匹配判断。对于两表连接，驱动表只需要被访问一次，而被驱动表需要访问多次。这个算法的开销非常大，复杂度为笛卡尔积。<br><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gf9s7d0psuj307406mmxc.jpg" alt=""><br>其实现伪代码如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">For each row r in R do -- 扫描R表（驱动表）</span><br><span class="line"> For each row s in S do -- 扫描S表（被驱动表）</span><br><span class="line"> If r and s satisfy the join condition -- 如果r和s满足join条件</span><br><span class="line"> Then output the tuple <r, s> -- 返回结果集</span><br></pre></td></tr></table></figure></p><a id="more"></a><h3 id="INLJ"><a href="#INLJ" class="headerlink" title="INLJ"></a>INLJ</h3><p>Index Nested-Loops Join(INLJ，基于索引的嵌套循环联接)，由于SNLJ算法非常粗暴，每次都会扫描全表。所以一般会在被驱动表的相关字段上建立索引，以降低SNLJ的复杂度，这种算法就被称为INLJ。<br>外表中的每条记录通过内表的索引进行访问，就是读取外部表一行数据，然后去内部表索引进行二分查找匹配；而一般B+树的高度为3~4层，也就是说匹配一次的io消耗也就3~4次，因此索引查询的成本是比较固定的，故优化器都倾向于使用记录数少的表作为外表。<br><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gf9sg65i6yj309n0a33z0.jpg" alt=""><br>其实现为代码如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">For each row r in R do -- 扫描R表</span><br><span class="line"> lookup s in S index -- 查询S表的索引（固定3~4次IO，B+树高度）</span><br><span class="line"> If find s == r -- 如果r匹配了索引s</span><br><span class="line"> Then output the tuple <r, s> -- 返回结果集</span><br></pre></td></tr></table></figure></p><h2 id="BNL"><a href="#BNL" class="headerlink" title="BNL"></a>BNL</h2><p>在SNLJ中，被驱动表需要进行全表扫描多次，由于MySQL的内存有限，所以可能会导致被驱动表中的数据页频繁的被读进内存又刷到磁盘中。<br>能不能有一个办法避免这样的情况发生呢？于是便加入了<code>join buffer</code>的概念。一次性判断多条记录，从而减少全表扫描的次数。<br>这种方式被称为BNL，BNL将外层循环的行/结果集存入到join buffer，然后每次遍历被驱动表都与join buffer中的数据进行比较。以此来减少全表扫描的次数。</p><p><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gf9smi9vqzj30a70873z3.jpg" alt=""><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">For each tuple r in R do -- 扫描外表R</span><br><span class="line"> store used columns as p from R in Join Buffer -- 将部分或者全部R的记录保存到Join Buffer中，记为p</span><br><span class="line"> For each tuple s in S do -- 扫描内表S</span><br><span class="line"> If p and s satisfy the join condition -- p与s满足join条件</span><br><span class="line"> Then output the tuple -- 返回为结果集</span><br></pre></td></tr></table></figure></p><p>开启BNL的方式：<code>set optimizer_switch='block_nested_loop=on'</code></p><h3 id="join-buffer"><a href="#join-buffer" class="headerlink" title="join buffer"></a>join buffer</h3><p>在MySQL实例中存在多种join查询的语句时，可通过调整join buffer的大小来减少磁盘访问。<br>但是由于join buffer是会话级别的，所以并不能设置太大，避免出现内存问题。</p><ul><li>系统变量Join_buffer_size决定了Join Buffer的大小。</li><li>Join Buffer可被用于联接是ALL、index、和range的类型。</li><li>每次联接使用一个Join Buffer，因此多表的联接可以使用多个Join Buffer。</li><li>Join Buffer在联接发生之前进行分配，在SQL语句执行完后进行释放。</li><li>Join Buffer只存储要进行查询操作的相关列数据，而不是整行的记录。</li></ul><h2 id="BKA"><a href="#BKA" class="headerlink" title="BKA"></a>BKA</h2><p>Batched Key Access Join(BKA，批量键访问联接)，MySQL在5.6版本中引入了BKA算法来对INLJ算法进行优化。<br>BKA其实就等价于MRR+INLJ。关于MRR的介绍可以参考<a href="https://omg-by.github.io/2020/05/27/new/MySQL/MySQL%E6%9C%BA%E5%88%B6%E4%BB%8B%E7%BB%8D%E4%B9%8BMRR/" target="_blank" rel="noopener">MySQL机制介绍之MRR</a></p><p><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gf9t27j1bhj30ki09p75j.jpg" alt=""><br>BKA主要工作流程为：</p><ol><li>将外部表中相关的列放入Join Buffer中。</li><li>批量的将Key（索引键值）发送到Multi-Range Read（MRR）接口。</li><li>Multi-Range Read（MRR）通过收到的Key，根据其对应的ROWID进行排序，然后再进行数据的读取操作。</li><li>返回结果集给客户端。</li></ol><p>开启BKA方式：<code>SET optimizer_switch='mrr=on,mrr_cost_based=off,batched_key_access=on';</code> </p><h2 id="CHJ"><a href="#CHJ" class="headerlink" title="CHJ"></a>CHJ</h2><p>// TODO studing</p><blockquote><p><a href="https://yq.aliyun.com/articles/27687" target="_blank" rel="noopener">https://yq.aliyun.com/articles/27687</a><br><a href="https://www.cnblogs.com/zuochanzi/p/10409752.html" target="_blank" rel="noopener">https://www.cnblogs.com/zuochanzi/p/10409752.html</a></p></blockquote>]]></content>
<summary type="html">
<h2 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h2><p>MySQL 5.5版本前，MySQL本身只支持一种表间关联方式，就是嵌套循环(Nested Loop)。如果关联表的数据量很大，则join关联的执行时间会非常长。<br>在5.5版本中，MySQL通过引入Block Nested-Loop Join(BNL)算法来优化嵌套执行。</p>
<h2 id="NLJ"><a href="#NLJ" class="headerlink" title="NLJ"></a>NLJ</h2><p>将驱动表的结果集作为循环基础数据，然后循环从该结果集中每次一条获取数据作为下一个表的过滤条件查询数据，然后合并结果。<br>如果有多表join，则将前面的表的结果集作为循环数据，取到每行再到链接的下一个表循环匹配。</p>
<h3 id="SNLJ"><a href="#SNLJ" class="headerlink" title="SNLJ"></a>SNLJ</h3><p>Simple Nested-Loops Join(SNLJ，简单嵌套循环联接)，该算法比较简单、直接。驱动表中的每一条记录都与被驱动表中的记录进行匹配判断。对于两表连接，驱动表只需要被访问一次，而被驱动表需要访问多次。这个算法的开销非常大，复杂度为笛卡尔积。<br><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gf9s7d0psuj307406mmxc.jpg" alt=""><br>其实现伪代码如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">For each row r in R do -- 扫描R表（驱动表）</span><br><span class="line"> For each row s in S do -- 扫描S表（被驱动表）</span><br><span class="line"> If r and s satisfy the join condition -- 如果r和s满足join条件</span><br><span class="line"> Then output the tuple &lt;r, s&gt; -- 返回结果集</span><br></pre></td></tr></table></figure></p>
</summary>
<category term="MySQL" scheme="http://yoursite.com/categories/MySQL/"/>
<category term="MySQL机制" scheme="http://yoursite.com/tags/MySQL%E6%9C%BA%E5%88%B6/"/>
</entry>
<entry>
<title>MySQL机制介绍之ICP</title>
<link href="http://yoursite.com/2020/05/28/new/MySQL/MySQL%E6%9C%BA%E5%88%B6%E4%BB%8B%E7%BB%8D%E4%B9%8BICP/"/>
<id>http://yoursite.com/2020/05/28/new/MySQL/MySQL机制介绍之ICP/</id>
<published>2020-05-28T05:22:48.000Z</published>
<updated>2020-05-29T00:45:48.291Z</updated>
<content type="html"><![CDATA[<h2 id="ICP是什么"><a href="#ICP是什么" class="headerlink" title="ICP是什么"></a>ICP是什么</h2><p>Index Condition Pushdown，也称为索引条件下推，体现在执行计划中会出现<code>Using index condition</code>。<br>ICP优化适用于MySQL利用索引从表里检索数据的场景。<br>使用命令<code>set optimizer_switch='index_condition_pushdown=on'</code>开启</p><h2 id="使用场景"><a href="#使用场景" class="headerlink" title="使用场景"></a>使用场景</h2><ul><li>索引访问方式是range/ref/eq_ref/ref_or_null，并且需要访问表的完整行记录</li><li>InnoDB和MYISAM表，包括分区的表(5.7)</li><li>对于InnoDB表，ICP只适用于二级索引。ICP的目标是减少访问表的完整行的读取量从而减少IO操作。</li><li>不支持建立在虚拟列上的二级索引</li><li>引用子查询、存储函数的条件没法下推</li><li>Triggered conditions 也没法下推</li></ul><h2 id="原理"><a href="#原理" class="headerlink" title="原理"></a>原理</h2><h3 id="不使用ICP"><a href="#不使用ICP" class="headerlink" title="不使用ICP"></a>不使用ICP</h3><ol><li>用二级索引查找数据的主键</li><li>用主键回表读取完整的行记录</li><li>引擎层利用where语句的条件对行记录进行过滤<br><img src="https://tva1.sinaimg.cn/large/007S8ZIlly1gf83cpvaofj30gw09bq3i.jpg" alt=""></li></ol><h3 id="使用ICP"><a href="#使用ICP" class="headerlink" title="使用ICP"></a>使用ICP</h3><ol><li>用二级索引查找数据的主键</li><li>如果where条件中的字段在复合索引中，引擎层对where条件里的字段进行过滤后，返回主键</li><li>利用主键回表读取完整的行记录</li><li>引擎层用where语句的剩余条件对行记录进行过滤<br><img src="https://tva1.sinaimg.cn/large/007S8ZIlly1gf83ct0w3xj30ge09b74t.jpg" alt=""></li></ol><h2 id="结论"><a href="#结论" class="headerlink" title="结论"></a>结论</h2><p>ICP的优化在引擎层就能够过滤掉大量的数据，这样无疑能够减少了对base table和mysql server的访问次数，提升了性能。</p><blockquote><p><a href="https://dev.mysql.com/doc/refman/5.6/en/index-condition-pushdown-optimization.html" target="_blank" rel="noopener">https://dev.mysql.com/doc/refman/5.6/en/index-condition-pushdown-optimization.html</a><br><a href="https://yq.aliyun.com/articles/259696" target="_blank" rel="noopener">https://yq.aliyun.com/articles/259696</a><br><a href="https://zhuanlan.zhihu.com/p/73035620" target="_blank" rel="noopener">https://zhuanlan.zhihu.com/p/73035620</a></p></blockquote>]]></content>
<summary type="html">
<h2 id="ICP是什么"><a href="#ICP是什么" class="headerlink" title="ICP是什么"></a>ICP是什么</h2><p>Index Condition Pushdown，也称为索引条件下推，体现在执行计划中会出现<code>Us
</summary>
<category term="MySQL" scheme="http://yoursite.com/categories/MySQL/"/>
<category term="MySQL机制" scheme="http://yoursite.com/tags/MySQL%E6%9C%BA%E5%88%B6/"/>
</entry>
<entry>
<title>MySQL机制介绍之MRR</title>
<link href="http://yoursite.com/2020/05/27/new/MySQL/MySQL%E6%9C%BA%E5%88%B6%E4%BB%8B%E7%BB%8D%E4%B9%8BMRR/"/>
<id>http://yoursite.com/2020/05/27/new/MySQL/MySQL机制介绍之MRR/</id>
<published>2020-05-27T04:59:51.000Z</published>
<updated>2020-05-29T00:45:31.813Z</updated>
<content type="html"><![CDATA[<h2 id="什么是MRR"><a href="#什么是MRR" class="headerlink" title="什么是MRR"></a>什么是MRR</h2><p>Multi-Range Read Optimization，是优化器将随机IO转换成顺序IO以降低查询过程中IO开销的一种手段。<br>它的好处有：</p><ul><li>使数据访问由随机变为顺序</li><li>减少缓冲池中页被替换的次数</li><li>批量处理查询操作</li></ul><p>可以通过<code>set optimizer_switch='mrr=on';</code>命令进行开启。</p><h3 id="不使用MRR"><a href="#不使用MRR" class="headerlink" title="不使用MRR"></a>不使用MRR</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">mysql> set optimizer_switch='mrr=off';</span><br><span class="line">Query OK, 0 rows affected (0.00 sec)</span><br><span class="line"></span><br><span class="line">mysql> explain select * from test.t1 where (a between 1 and 10) and (c between 9 and 10) ;</span><br><span class="line">+----+-------------+-------+-------+---------------+------+---------+------+------+------------------------------------+</span><br><span class="line">| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |</span><br><span class="line">+----+-------------+-------+-------+---------------+------+---------+------+------+------------------------------------+</span><br><span class="line">| 1 | SIMPLE | t1 | range | mrrx,xx | xx | 5 | NULL | 2 | Using index condition; Using where |</span><br><span class="line">+----+-------------+-------+-------+---------------+------+---------+------+------+------------------------------------+</span><br><span class="line">1 row in set (0.00 sec)</span><br></pre></td></tr></table></figure><p>在不使用MRR时，优化器需要根据二级索引返回的记录来进行回表，这个过程一般会有较多的随机IO操作。</p><h3 id="使用MRR"><a href="#使用MRR" class="headerlink" title="使用MRR"></a>使用MRR</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">mysql> set optimizer_switch='mrr=on';</span><br><span class="line">Query OK, 0 rows affected (0.00 sec)</span><br><span class="line"></span><br><span class="line">mysql> explain select * from test.t1 where (a between 1 and 10) and (c between 9 and 10) ;</span><br><span class="line">+----+-------------+-------+-------+---------------+------+---------+------+------+-----------------------------------------------+</span><br><span class="line">| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |</span><br><span class="line">+----+-------------+-------+-------+---------------+------+---------+------+------+-----------------------------------------------+</span><br><span class="line">| 1 | SIMPLE | t1 | range | mrrx,xx | xx | 5 | NULL | 2 | Using index condition; Using where; Using MRR |</span><br><span class="line">+----+-------------+-------+-------+---------------+------+---------+------+------+-----------------------------------------------+</span><br><span class="line">1 row in set (0.00 sec)</span><br></pre></td></tr></table></figure><p>在使用了MRR时，SQL语句的执行过程为：</p><ol><li>优化器将二级索引查询到的记录放到一块缓冲区中(read_rnd_buffer_size)</li><li>如果二级索引扫描到文件末尾或者缓冲区已满，则使用快排对缓冲区中的内容按照主键进行排序</li><li>用户线程调用MRR接口获取cluster index，然后根据cluster index获取行数据</li><li>当缓冲区中的cluster index取完数据，则继续调用过程2、3，直到扫描结束</li></ol><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>MRR特性就是在查询过程中，先将满足条件的id查询出来并进行排序后，再进行批量查询操作。从而实现随机IO到顺序IO的转换，提升性能。</p><blockquote><p><a href="https://dev.mysql.com/doc/refman/5.6/en/mrr-optimization.html" target="_blank" rel="noopener">https://dev.mysql.com/doc/refman/5.6/en/mrr-optimization.html</a><br><a href="https://zhuanlan.zhihu.com/p/110154066" target="_blank" rel="noopener">https://zhuanlan.zhihu.com/p/110154066</a></p></blockquote>]]></content>
<summary type="html">
<h2 id="什么是MRR"><a href="#什么是MRR" class="headerlink" title="什么是MRR"></a>什么是MRR</h2><p>Multi-Range Read Optimization，是优化器将随机IO转换成顺序IO以降低查询过程中
</summary>
<category term="MySQL" scheme="http://yoursite.com/categories/MySQL/"/>
<category term="MySQL机制" scheme="http://yoursite.com/tags/MySQL%E6%9C%BA%E5%88%B6/"/>
</entry>
<entry>
<title>Redis扩展命令实现</title>
<link href="http://yoursite.com/2020/05/17/new/Redis/Redis%E6%89%A9%E5%B1%95%E5%91%BD%E4%BB%A4%E5%AE%9E%E7%8E%B0/"/>
<id>http://yoursite.com/2020/05/17/new/Redis/Redis扩展命令实现/</id>
<published>2020-05-17T05:41:47.000Z</published>
<updated>2020-05-17T06:06:30.394Z</updated>
<content type="html"><![CDATA[<p>上周，参加公司后端部门开发的分享。<br>在期间，该开发吐槽Redis好几个使用不便利的地方。</p><ol><li><font color="red">Redis没有批量设置过期时间的命令</font></li><li><font color="red">incr 不存在的key的时候，并不会设置过期时间。导致持久化key的存在</font></li></ol><p><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gevcwyswasj30uk0l20wd.jpg" alt=""></p><a id="more"></a><p>作为客户端，解决办法只有使用lua脚本来进行实现：<br><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gevcuiy2fbj30zk09g0v0.jpg" alt=""></p><h2 id="思考"><a href="#思考" class="headerlink" title="思考"></a>思考</h2><p>虽然说使用lua脚本也能够解决这样的问题，但是对用户体验不太友好，同时也增加了编码的复杂度。<br>而且这样的功能实现起来并不算复杂，为什么不可以在Redis服务端去实现这样的功能呢？</p><h2 id="实现"><a href="#实现" class="headerlink" title="实现"></a>实现</h2><p>说干咱就干。<br>利用周末的时间，简单的实现了其中的一个槽点功能:mexpire<br>可能会存在考虑欠缺的地方，但基本功能还是实现了的。</p><h3 id="expire-c"><a href="#expire-c" class="headerlink" title="expire.c"></a>expire.c</h3><p><code>expire.c</code>主要是对过期管理的文件。<br>一开始只想实现mexpire功能，结果发现还有<code>expireta</code>、<code>pexpire</code>、<code>pexpirate</code>命令跟<code>expire</code>命令相近，并且底层实现都是同一个函数。<br>于是就一起实现了。</p><p>逻辑其实很简单，就是遍历传过来的参数。</p><ul><li>如果key存在，就设置过期时间。并计数。</li><li>如果key不存在，就跳过。</li><li>返回成功设置过期时间个数。</li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br></pre></td><td class="code"><pre><span class="line">void mexpireGenericCommand(client *c, long long basetime, int unit){</span><br><span class="line"></span><br><span class="line"> if ((c->argc % 2) == 0){</span><br><span class="line"> addReplyErrorFormat(c, "wrong number of arguments for %s", c->cmd->name);</span><br><span class="line"> return;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> int j;</span><br><span class="line"> robj *key, *param;</span><br><span class="line"> long long when;</span><br><span class="line"> int nums = 0;</span><br><span class="line"> for (j = 1; j < c->argc; j+=2){</span><br><span class="line"> key = c->argv[j];</span><br><span class="line"> param = c->argv[j+1];</span><br><span class="line"> if ((getLongLongFromObject(param, &when) != C_OK) || lookupKeyWrite(c->db,key) == NULL){</span><br><span class="line"> continue;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> if (unit == UNIT_SECONDS) when *= 1000;</span><br><span class="line"> when += basetime;</span><br><span class="line"></span><br><span class="line"> nums++;</span><br><span class="line"> if (when <= mstime() && !server.loading && !server.masterhost) {</span><br><span class="line"> int deleted = server.lazyfree_lazy_expire ? dbAsyncDelete(c->db,key) :</span><br><span class="line"> dbSyncDelete(c->db,key);</span><br><span class="line"> serverAssertWithInfo(c,key,deleted);</span><br><span class="line"> server.dirty++;</span><br><span class="line"></span><br><span class="line"> signalModifiedKey(c->db,key);</span><br><span class="line"> notifyKeyspaceEvent(NOTIFY_GENERIC,"del",key,c->db->id);</span><br><span class="line"></span><br><span class="line"> } else {</span><br><span class="line"> setExpire(c,c->db,key,when);</span><br><span class="line"> signalModifiedKey(c->db,key);</span><br><span class="line"> notifyKeyspaceEvent(NOTIFY_GENERIC,"expire",key,c->db->id);</span><br><span class="line"> server.dirty++;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> addReplyLongLong(c, nums);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">void mexpireCommand(client *c){</span><br><span class="line"> mexpireGenericCommand(c, mstime(), UNIT_SECONDS);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">void mexpireatCommand(client *c) {</span><br><span class="line"> mexpireGenericCommand(c,0,UNIT_SECONDS);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">void mpexpireCommand(client *c) {</span><br><span class="line"> mexpireGenericCommand(c,mstime(),UNIT_MILLISECONDS);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">void mpexpireatCommand(client *c) {</span><br><span class="line"> mexpireGenericCommand(c,0,UNIT_MILLISECONDS);</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="server-h"><a href="#server-h" class="headerlink" title="server.h"></a>server.h</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">void mexpireCommand(client *c);</span><br><span class="line">void mexpireatCommand(client *c);</span><br><span class="line">void mpexpireCommand(client *c);</span><br><span class="line">void mpexpireatCommand(client *c);</span><br></pre></td></tr></table></figure><h3 id="server-c"><a href="#server-c" class="headerlink" title="server.c"></a>server.c</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">{"mexpire",mexpireCommand,-3, "write @keyspace",0,NULL,1,-1,2,0,0,0},</span><br><span class="line">{"mexpireat",mexpireatCommand,-3,"write @keyspace", 0,NULL,1,-1,2,0,0,0},</span><br><span class="line">{"mpexpire",mpexpireCommand,-3, "write @keyspace", 0,NULL,1,-1,2,0,0,0},</span><br><span class="line">{"mpexpireat",mpexpireatCommand,-3, "write @keyspace", 0,NULL,1,-1,2,0,0,0},</span><br></pre></td></tr></table></figure><p>这里有必要说明一下上面配置的解释。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">struct redisCommand {</span><br><span class="line"> char *name;</span><br><span class="line"> redisCommandProc *proc;</span><br><span class="line"> int arity;</span><br><span class="line"> char *sflags; /* Flags as string representation, one char per flag. */</span><br><span class="line"> uint64_t flags; /* The actual flags, obtained from the 'sflags' field. */</span><br><span class="line"> /* Use a function to determine keys arguments in a command line.</span><br><span class="line"> * Used for Redis Cluster redirect. */</span><br><span class="line"> redisGetKeysProc *getkeys_proc;</span><br><span class="line"> /* What keys should be loaded in background when calling this command? */</span><br><span class="line"> int firstkey; /* The first argument that's a key (0 = no keys) */</span><br><span class="line"> int lastkey; /* The last argument that's a key */</span><br><span class="line"> int keystep; /* The step between first and last key */</span><br><span class="line"> long long microseconds, calls;</span><br><span class="line"> int id; /* Command ID. This is a progressive ID starting from 0 that</span><br><span class="line"> is assigned at runtime, and is used in order to check</span><br><span class="line"> ACLs. A connection is able to execute a given command if</span><br><span class="line"> the user associated to the connection has this command</span><br><span class="line"> bit set in the bitmap of allowed commands. */</span><br><span class="line">};</span><br></pre></td></tr></table></figure></p><ul><li>name：命令名称</li><li>function：指向函数</li><li>arity：参数限制</li><li>sflags：命令属性</li><li>flags：命令属性掩码，一般为0</li><li>get_keys_proc：在复杂参数下，指定那个才是真正的key。一般为NULL</li><li>first_key_index：第一个参数所在位置</li><li>last_key_index：最后一个参数所在位置</li><li>key_step：命令步长</li><li>microseconds：命令的度量项，由Redis来设置，并且总是初始化为0。</li><li>calls：命令的度量项，由Redis来设置，并且总是初始化为0。</li><li>id：命令的权限，由Redis来设置，并且总是初始化为0。</li></ul><p>只需要修改以上几个文件然后启动就可以了。是不是很简单？<br>PS：前一篇文章中解决哨兵BUG<a href="https://omg-by.github.io/2020/05/17/new/redis_BUG%E8%AE%B0%E5%BD%95%E4%B8%80%E5%88%99/" target="_blank" rel="noopener">《Redis哨兵client-reconfig-script脚本bug记录一则》</a>时，调试就需要重新进行make && make install操作才行。</p><h2 id="使用"><a href="#使用" class="headerlink" title="使用"></a>使用</h2><p>命令使用，跟正常使用其他命令并没有什么太大区别。主要会跟<code>mset</code>命令使用比较相似。<br><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gevdm6iviij30la0e8gsd.jpg" alt=""></p><h2 id="issue"><a href="#issue" class="headerlink" title="issue"></a>issue</h2><p>本来还想去github提提issue的，结果发现早就有人跟作者提过这些问题了。我还是too yong to simple呀。<br>但是本着学习的心态，还是提了个issue，问问作者为什么不去实现这些简单又好用的命令。<a href="https://github.com/antirez/redis/issues/7263" target="_blank" rel="noopener">https://github.com/antirez/redis/issues/7263</a><br>后面有时间还是会继续实现其他命令的。</p>]]></content>
<summary type="html">
<p>上周，参加公司后端部门开发的分享。<br>在期间，该开发吐槽Redis好几个使用不便利的地方。</p>
<ol>
<li><font color="red">Redis没有批量设置过期时间的命令</font></li>
<li><font color="red">incr 不存在的key的时候，并不会设置过期时间。导致持久化key的存在</font>
</li>
</ol>
<p><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gevcwyswasj30uk0l20wd.jpg" alt=""></p>
</summary>
<category term="Redis" scheme="http://yoursite.com/categories/Redis/"/>
<category term="Redis" scheme="http://yoursite.com/tags/Redis/"/>
</entry>
<entry>
<title>MySQL闪回工具调研</title>
<link href="http://yoursite.com/2020/05/17/new/MySQL/MySQL%E9%97%AA%E5%9B%9E%E5%B7%A5%E5%85%B7/"/>
<id>http://yoursite.com/2020/05/17/new/MySQL/MySQL闪回工具/</id>
<published>2020-05-16T16:41:47.000Z</published>
<updated>2020-06-13T16:00:45.240Z</updated>
<content type="html"><![CDATA[<p>刚入职2个月左右的时候，就遇到了业务误操作，将对测试环境的delete操作，到线上执行了。。。(至于业务为什么有delete权限的账号，俺也不知道)<br>由于公司所有实例都部署在阿里云上，所以只能依赖于阿里云的备份恢复系统来进行数据恢复。但是非常慢，当时是大概花了20分钟才完全恢复（-_-||）。<br>于是乎~工作量又被增加了。领导让我调研数据闪回工具。。。</p><p>在工具调研测试过程中，发现了这几个工具存在部分BUG问题，所以给记录一下。避免再次踩坑。</p><h2 id="MyFlash"><a href="#MyFlash" class="headerlink" title="MyFlash"></a>MyFlash</h2><p>MyFlash是由美团点评公司技术工程部开发维护的一个回滚DML操作的工具。该工具通过解析v4版本的binlog，完成回滚操作。相对已有的回滚工具，其增加了更多的过滤选项，让回滚更加容易。</p><font color="red">通过解析binlog来生成回滚binlog文件。</font><h3 id="优缺点"><a href="#优缺点" class="headerlink" title="优缺点"></a>优缺点</h3><p>优点：</p><ul><li>多种过滤条件，能够按照需求实现精准过滤</li><li>支持离线解析。不会对运行实例造成影响</li></ul><p>缺点：</p><ul><li>binlog格式必须为row，并且binlog_row_image=full。</li><li>只支持5.6和5.7</li><li>只支持DML，不支持DDL</li><li>MyFlash不能解析阿里云RDS的binlog</li></ul><a id="more"></a> <h3 id="安装"><a href="#安装" class="headerlink" title="安装"></a>安装</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"># 下载源代码</span><br><span class="line">git clone https://github.com/Meituan-Dianping/MyFlash.git</span><br><span class="line"> </span><br><span class="line"># 编译</span><br><span class="line">cd MyFlash</span><br><span class="line">sh build.sh</span><br></pre></td></tr></table></figure><h3 id="使用"><a href="#使用" class="headerlink" title="使用"></a>使用</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">[root@node001 binary]# ./flashback --help</span><br><span class="line">Usage:</span><br><span class="line">flashback [OPTION...]</span><br><span class="line"> </span><br><span class="line">Help Options:</span><br><span class="line">-h, --help Show help options</span><br><span class="line"> </span><br><span class="line">Application Options:</span><br><span class="line">--databaseNames databaseName to apply. if multiple, seperate by comma(,) 库名，多个库之间用“，”分割</span><br><span class="line">--tableNames tableName to apply. if multiple, seperate by comma(,) 表名，多个表之间用“，”分割</span><br><span class="line">--start-position start position 开始的位点。如不指定，从文件的开始处回滚。</span><br><span class="line">--stop-position stop position 结束的位点。如不指定，回滚到文件结尾。</span><br><span class="line">--start-datetime start time (format %Y-%m-%d %H:%M:%S) 开始的时间点。如不指定，则不限定时间</span><br><span class="line">--stop-datetime stop time (format %Y-%m-%d %H:%M:%S) 结束的时间点。如不指定，则不限定时间</span><br><span class="line">--sqlTypes sql type to filter . support INSERT, UPDATE ,DELETE. if multiple, seperate by comma(,) 需要回滚的SQL类型，只支持INSERT, UPDATE,DELETE。多个类型之间以“，”分割</span><br><span class="line">--maxSplitSize max file size after split, the uint is M 生成文件分割大小。以M为单位</span><br><span class="line">--binlogFileNames binlog files to process. if multiple, seperate by comma(,) binlog文件名，支持多个文件</span><br><span class="line">--outBinlogFileNameBase output binlog file name base 输出文件名前缀。文件名后缀为：.flashback</span><br><span class="line">--logLevel log level, available option is debug,warning,error</span><br><span class="line">--include-gtids gtids to process 生成的语句包含gtid</span><br><span class="line">--exclude-gtids gtids to skip 跳过gtid</span><br></pre></td></tr></table></figure><h2 id="binlog2sql"><a href="#binlog2sql" class="headerlink" title="binlog2sql"></a>binlog2sql</h2><font color="red">通过模拟从库解析数据来生成回滚SQL。</font><h3 id="优缺点-1"><a href="#优缺点-1" class="headerlink" title="优缺点"></a>优缺点</h3><p>优点：</p><ul><li>直接生成回滚语句，可读性较好。可根据需求进行选择性回滚</li><li>python实现，安装、使用、可扩展性较好</li><li>可指定位点、时间、语句类型</li><li>支持阿里云、自建实例</li></ul><p>缺点：</p><ul><li>需要连接数据库读取binlog，会对线上环境造成一定负载</li><li>依赖binlog。如果binlog被清理则无法生成回滚语句</li><li>不支持离线解析</li><li>解析速度较慢</li></ul><h3 id="安装使用"><a href="#安装使用" class="headerlink" title="安装使用"></a>安装使用</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line">shell> git clone https://github.com/danfengcao/binlog2sql.git && cd binlog2sql</span><br><span class="line">shell> pip install -r requirements.txt</span><br><span class="line"></span><br><span class="line">用户授权</span><br><span class="line">GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO</span><br><span class="line"></span><br><span class="line">使用</span><br><span class="line">mysql连接配置</span><br><span class="line">-h host; -P port; -u user; -p password</span><br><span class="line"> </span><br><span class="line">解析模式</span><br><span class="line">--stop-never 持续解析binlog。可选。默认False，同步至执行命令时最新的binlog位置。</span><br><span class="line">-K, --no-primary-key 对INSERT语句去除主键。可选。默认False</span><br><span class="line">-B, --flashback 生成回滚SQL，可解析大文件，不受内存限制。可选。默认False。与stop-never或no-primary-key不能同时添加。</span><br><span class="line">--back-interval -B模式下，每打印一千行回滚SQL，加一句SLEEP多少秒，如不想加SLEEP，请设为0。可选。默认1.0。</span><br><span class="line"> </span><br><span class="line">解析范围控制</span><br><span class="line">--start-file 起始解析文件，只需文件名，无需全路径 。必须。</span><br><span class="line">--start-position/--start-pos 起始解析位置。可选。默认为start-file的起始位置。</span><br><span class="line">--stop-file/--end-file 终止解析文件。可选。默认为start-file同一个文件。若解析模式为stop-never，此选项失效。</span><br><span class="line">--stop-position/--end-pos 终止解析位置。可选。默认为stop-file的最末位置；若解析模式为stop-never，此选项失效。</span><br><span class="line">--start-datetime 起始解析时间，格式'%Y-%m-%d %H:%M:%S'。可选。默认不过滤。</span><br><span class="line">--stop-datetime 终止解析时间，格式'%Y-%m-%d %H:%M:%S'。可选。默认不过滤。</span><br><span class="line"> </span><br><span class="line">对象过滤</span><br><span class="line">-d, --databases 只解析目标db的sql，多个库用空格隔开，如-d db1 db2。可选。默认为空。</span><br><span class="line">-t, --tables 只解析目标table的sql，多张表用空格隔开，如-t tbl1 tbl2。可选。默认为空。</span><br><span class="line">--only-dml 只解析dml，忽略ddl。可选。默认False。</span><br><span class="line">--sql-type 只解析指定类型，支持INSERT, UPDATE, DELETE。多个类型用空格隔开，如--sql-type INSERT DELETE。可选。默认为增删改都解析。用了此参数但没填任何类型，则三者都不解析。</span><br></pre></td></tr></table></figure><h3 id="发现的问题"><a href="#发现的问题" class="headerlink" title="发现的问题"></a>发现的问题</h3><h4 id="datetime类型恢复错误"><a href="#datetime类型恢复错误" class="headerlink" title="datetime类型恢复错误"></a>datetime类型恢复错误</h4><p>问题描述：</p><ul><li>存在字段：<code>o_update_time</code> datetime NOT NULL COMMENT</li><li>存在值：o_update_time: 0000-00-00 00:00:00</li><li>问题一：删除后，binlog2sql生成的insert语句，会将该字段值给设置为NULL。导致插入报错失败。</li><li>问题二：阿里云恢复后，数据不一致。</li></ul><p>删除前数据：<br><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gfr3r95zxhj319i0u04ak.jpg" alt=""></p><p>binlog2sql生成语句：<br><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1gfr3sz7dg2j32hc09o79o.jpg" alt=""></p><h4 id="varbinary类型字段恢复错误"><a href="#varbinary类型字段恢复错误" class="headerlink" title="varbinary类型字段恢复错误"></a>varbinary类型字段恢复错误</h4><p>现数据库中存在经过加密函数加密后的字段。并存在几条数据。然后删除其中一条语句来测试工具生成回滚语句。<br><code>pwd</code> varbinary(255) NOT NULL COMMENT ‘密码’,<br><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1geurfbnj9fj30jr06c74x.jpg" alt=""></p><p>使用binlog2sql生成回滚语句时出错<br><img src="https://tva1.sinaimg.cn/large/007S8ZIlgy1geurfewn49j30ps03b0tc.jpg" alt=""></p><h4 id="精度问题"><a href="#精度问题" class="headerlink" title="精度问题"></a>精度问题</h4><p>在数据类型为decimal、double、float等精度类型时，恢复时会由于精度问题导致恢复数据不一致。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">mysql> show create table huzb_deci\G</span><br><span class="line">*************************** 1. row ***************************</span><br><span class="line"> Table: huzb_deci</span><br><span class="line">Create Table: CREATE TABLE `huzb_deci` (</span><br><span class="line"> `id` int(11) NOT NULL DEFAULT '0',</span><br><span class="line"> `dec` decimal(5,2) DEFAULT NULL,</span><br><span class="line"> `flo` float DEFAULT NULL,</span><br><span class="line"> `dou` double DEFAULT NULL,</span><br><span class="line"> PRIMARY KEY (`id`)</span><br><span class="line">) ENGINE=InnoDB DEFAULT CHARSET=utf8</span><br><span class="line">1 row in set (0.00 sec)</span><br></pre></td></tr></table></figure></p><p>插入几条数据后，使用binlog2sql进行恢复。可以发现存在以下问题：</p><ol><li>double类型在插入时，由于精度问题，插入数据与原数据已经不一致。</li><li>float类型数据在使用binlog2sql生成恢复语句时，转换成了double类型，小数点后多了几位精度。</li><li>生成的回滚语句执行后，恢复的double类型数据不一致。</li></ol><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line">mysql> insert into huzb_deci values(1,1.1111111111111111111111111111111,2.22222222,3.333333333333333333333);</span><br><span class="line">Query OK, 1 row affected, 1 warning (0.00 sec)</span><br><span class="line"> </span><br><span class="line">mysql> show warnings;</span><br><span class="line">+-------+------+------------------------------------------+</span><br><span class="line">| Level | Code | Message |</span><br><span class="line">+-------+------+------------------------------------------+</span><br><span class="line">| Note | 1265 | Data truncated for column 'dec' at row 1 |</span><br><span class="line">+-------+------+------------------------------------------+</span><br><span class="line">1 row in set (0.01 sec)</span><br><span class="line"> </span><br><span class="line">mysql> select * from huzb_deci;</span><br><span class="line">+----+------+---------+--------------------+</span><br><span class="line">| id | dec | flo | dou |</span><br><span class="line">+----+------+---------+--------------------+</span><br><span class="line">| 1 | 1.11 | 2.22222 | 3.3333333333333335 |</span><br><span class="line">+----+------+---------+--------------------+</span><br><span class="line">1 row in set (0.01 sec)</span><br><span class="line"> </span><br><span class="line"> </span><br><span class="line"># 利用binlog2sql生成恢复语句并执行。</span><br><span class="line">mysql> INSERT INTO `huzb`.`huzb_deci`(`id`, `dec`, `flo`, `dou`) VALUES (1, 1.11, 2.22222232818604, 3.33333333333333); #start 260805 end 260989 time 2019-10-30 19:58:48</span><br><span class="line">Query OK, 1 row affected (0.00 sec)</span><br><span class="line"> </span><br><span class="line">mysql> select * from huzb_deci;</span><br><span class="line">+----+------+---------+------------------+</span><br><span class="line">| id | dec | flo | dou |</span><br><span class="line">+----+------+---------+------------------+</span><br><span class="line">| 1 | 1.11 | 2.22222 | 3.33333333333333 |</span><br><span class="line">+----+------+---------+------------------+</span><br><span class="line">1 row in set (0.01 sec)</span><br></pre></td></tr></table></figure><h3 id="问题修复"><a href="#问题修复" class="headerlink" title="问题修复"></a>问题修复</h3><h4 id="日期错误问题修复"><a href="#日期错误问题修复" class="headerlink" title="日期错误问题修复"></a>日期错误问题修复</h4><ol><li>按照<a href="https://github.com/noplay/python-mysql-replication/pull/228修改对应的python-mysql-replication包文件(需要修改：binlogstream.py和row_event.py两个文件)" target="_blank" rel="noopener">https://github.com/noplay/python-mysql-replication/pull/228修改对应的python-mysql-replication包文件(需要修改：binlogstream.py和row_event.py两个文件)</a></li><li>binlog2sql中修改。增加date_tostr参数</li><li>不能直接使用python-mysql-replication的最新依赖包。只能通过修改文件，可能会引起未知问题</li><li>如果无NULL值字段，可尝试使用文本替换方式来修改插入语句。<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">binlog2sql.py 修改：</span><br><span class="line">stream = BinLogStreamReader(connection_settings=self.conn_setting, server_id=self.server_id,</span><br><span class="line"> log_file=self.start_file, log_pos=self.start_pos, only_schemas=self.only_schemas,</span><br><span class="line"> only_tables=self.only_tables, resume_stream=True, blocking=True, date_tostr=True)</span><br></pre></td></tr></table></figure></li></ol><h4 id="二进制流数据导入导出错误问题修复"><a href="#二进制流数据导入导出错误问题修复" class="headerlink" title="二进制流数据导入导出错误问题修复"></a>二进制流数据导入导出错误问题修复</h4><p>根据mysqldump导出二进制数据时的方式，所以有以下修复思路：</p><ol><li>在生成sql文本时，将二进制流转换成十六进制。</li><li>在导入数据时，利用原生的unhex将十六进制转换成二进制。<br>修改binlog2sql_util.py以下位置:<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"># TODO 改动一：二进制流解析为十六进制</span><br><span class="line">def fix_object(value):</span><br><span class="line"> """Fixes python objects so that they can be properly inserted into SQL queries"""</span><br><span class="line"> if isinstance(value, set):</span><br><span class="line"> value = ','.join(value)</span><br><span class="line"> if PY3PLUS and isinstance(value, bytes):</span><br><span class="line"> # return value.decode('utf-8', errors='ignore')</span><br><span class="line"> # 将二进制数据转换成十六进制数据，并使用特殊符号包含起来。用于后面转换</span><br><span class="line"> return "----" + bytes.hex(value) + "____"</span><br><span class="line"> elif not PY3PLUS and isinstance(value, unicode):</span><br><span class="line"> return value.encode('utf-8')</span><br><span class="line"> else:</span><br><span class="line"> return value</span><br><span class="line"> </span><br><span class="line"> </span><br><span class="line"># TODO 改动二：修改生成sql中的unhex串</span><br><span class="line">unhex_inex = str.find(sql, "'----")</span><br><span class="line">if unhex_inex != -1:</span><br><span class="line"> sql = str.replace(sql, "'----", "unhex('")</span><br><span class="line"> sql = str.replace(sql, "____'", "')")</span><br><span class="line">sql += ' #start %s end %s time %s' % (e_start_pos, binlog_event.packet.log_pos, time)</span><br></pre></td></tr></table></figure></li></ol><p>题外话：<br>小公司还是各种体系、规范不够完善，要是在上一家公司，早就被业务吊起来diss不知道多少遍了。<br>数据备份恢复是作为DBA最重要的技能点之一，但是由于云化的出现，导致很多人都只会点点点，太过于依赖平台的操作，而忽略了DBA的本质工作。这也是我接下来的工作重点：MySQL、Redis自动化备份恢复系统的建设工作（已完成）。</p>]]></content>
<summary type="html">
<p>刚入职2个月左右的时候，就遇到了业务误操作，将对测试环境的delete操作，到线上执行了。。。(至于业务为什么有delete权限的账号，俺也不知道)<br>由于公司所有实例都部署在阿里云上，所以只能依赖于阿里云的备份恢复系统来进行数据恢复。但是非常慢，当时是大概花了20分钟才完全恢复（-_-||）。<br>于是乎~工作量又被增加了。领导让我调研数据闪回工具。。。</p>
<p>在工具调研测试过程中，发现了这几个工具存在部分BUG问题，所以给记录一下。避免再次踩坑。</p>
<h2 id="MyFlash"><a href="#MyFlash" class="headerlink" title="MyFlash"></a>MyFlash</h2><p>MyFlash是由美团点评公司技术工程部开发维护的一个回滚DML操作的工具。该工具通过解析v4版本的binlog，完成回滚操作。相对已有的回滚工具，其增加了更多的过滤选项，让回滚更加容易。</p>
<font color="red">通过解析binlog来生成回滚binlog文件。</font>
<h3 id="优缺点"><a href="#优缺点" class="headerlink" title="优缺点"></a>优缺点</h3><p>优点：</p>
<ul>
<li>多种过滤条件，能够按照需求实现精准过滤</li>
<li>支持离线解析。不会对运行实例造成影响</li>
</ul>
<p>缺点：</p>
<ul>
<li>binlog格式必须为row，并且binlog_row_image=full。</li>
<li>只支持5.6和5.7</li>
<li>只支持DML，不支持DDL</li>
<li>MyFlash不能解析阿里云RDS的binlog</li>
</ul>
</summary>
<category term="MySQL" scheme="http://yoursite.com/categories/MySQL/"/>
</entry>
<entry>
<title>Redis哨兵client-reconfig-script脚本bug记录一则</title>
<link href="http://yoursite.com/2020/05/17/new/Redis/redis_BUG%E8%AE%B0%E5%BD%95%E4%B8%80%E5%88%99/"/>
<id>http://yoursite.com/2020/05/17/new/Redis/redis_BUG记录一则/</id>
<published>2020-05-16T16:00:47.000Z</published>
<updated>2020-05-16T16:06:27.092Z</updated>
<content type="html"><![CDATA[<p>前一阵子一直在做自建机房Redis主从的环境搭建。了解到哨兵高可用切换后的会调用<code>client-reconfig-script</code>参数配置的脚本。<br>但是遇到了一个从2.8版本一直存在至今的BUG。我已经提了一个PR给官方，并被meger了。<a href="https://github.com/antirez/redis/pull/7113" target="_blank" rel="noopener">https://github.com/antirez/redis/pull/7113</a><br>特此记录一下。</p><h2 id="BUG场景"><a href="#BUG场景" class="headerlink" title="BUG场景"></a>BUG场景</h2><p>手动将主实例kill掉，模拟宕机情况。在某些情况下，<font color="red">哨兵已经触发了高可用切换</font>行为（主从状态、日志均有）。但是并<font color="red">没有调用</font>配置的脚本（非必现，但是落到同一台机器调用时并不会调用）<br>重启该机器上的哨兵节点又恢复正常。（重启大法好）</p><a id="more"></a><h2 id="环境说明"><a href="#环境说明" class="headerlink" title="环境说明"></a>环境说明</h2><p>根据网上搜索到的脚本也自己编写了一个。大概逻辑就是</p><ol><li>新主IP等于本机IP触发域名切换、元数据修改等操作，并exit 0;</li><li>非本机IP不做任何操作，并exit 1;</li></ol><p>按道理来说，这个脚本处理逻辑跟网上99%给出的脚本一致，应该问题不大。<br>的确，在前几次或者短时间内触发多次触发高可用切换，脚本都能够正常执行。<br>但是遇到以下几种情况下不会触发。</p><ol><li>触发2次高可用切换后，本人划水半小时，再来触发，此时脚本不执行。</li><li>连续触发10次左右，都正常。总时长在5分钟左右后，脚本不执行。</li></ol><h2 id="问题排查"><a href="#问题排查" class="headerlink" title="问题排查"></a>问题排查</h2><h3 id="脚本问题？"><a href="#脚本问题？" class="headerlink" title="脚本问题？"></a>脚本问题？</h3><p>首先，由于是第一次接触哨兵调用脚本。所以怀疑是自己写的脚本逻辑不正确。于是在编写脚本中每一个操作前都输出日志，甚至在第一行输出东西；结果仍没有调用！<br>而且轮到其他机器上的哨兵调用脚本时，可能能够调用成功。<br>所以排除脚本问题。</p><h3 id="脚本权限问题？"><a href="#脚本权限问题？" class="headerlink" title="脚本权限问题？"></a>脚本权限问题？</h3><p>通过google在Stack Overflow上，以及在Redis交流群中咨询。了解到可能存在脚本权限问题可能会调用失败。<br>于是关注该脚本在每次被调用后的状态，发现并没有什么变化。并且机器为新机器，只有本人在操作。所以认为权限问题不大可能。</p><h3 id="发现共性"><a href="#发现共性" class="headerlink" title="发现共性"></a>发现共性</h3><p>在进行多次高可用切换测试后，所有的哨兵切换在执行完高可用切换后，都不再去调用脚本。<br>这个时候，对所有的哨兵节点状态进行查看。发现有一个共性。<code>sentinel_running_scripts</code>值都为16。<br>该参数表示正在执行的脚本。<br>进一步验证，发现：</p><ol><li>该值小于16时，会正常调用。</li><li>该值会进行周期性的增加。</li><li>只触发一次高可用时，该值变成9后不再增加。</li></ol><h2 id="源码分析"><a href="#源码分析" class="headerlink" title="源码分析"></a>源码分析</h2><font color="red">没有什么问题是阅读源码解决不了的</font><p>通过分析哨兵节点进行高可用切换段代码。发现在调用<code>client-reconfig-script</code>脚本时，会根据其返回值做不同处理。</p><ul><li>0：表示脚本执行成功。不重试</li><li>1：表示脚本执行失败。进行重试，最多10次</li><li>大于1：表示脚本执行失败。不进行重试。</li></ul><p>bug出现点：</p><ul><li>当running_scripts >= SENTINEL_SCRIPT_MAX_RUNNING(16)时就不会再进入到调用脚本的逻辑里。</li><li>当调用脚本时，running_scripts++</li><li>脚本重试也会触发running_scripts++</li><li>只有当脚本达到最大重试次数(10次)，或者脚本返回非1值时，才触发一次running_scripts–</li></ul><p>可以看到，在非新主机器脚本执行时，脚本总会exit 1。所以会重试10次。running_scripts+10-1=9。<br>当遇到两次这样的情况，running_scripts就等于16了。调用脚本逻辑将不再被执行。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br></pre></td><td class="code"><pre><span class="line">/* Run pending scripts if we are not already at max number of running</span><br><span class="line"> * scripts. */</span><br><span class="line">void sentinelRunPendingScripts(void) {</span><br><span class="line"> listNode *ln;</span><br><span class="line"> listIter li;</span><br><span class="line"> mstime_t now = mstime();</span><br><span class="line"></span><br><span class="line"> /* Find jobs that are not running and run them, from the top to the</span><br><span class="line"> * tail of the queue, so we run older jobs first. */</span><br><span class="line"> // li是script_queue的一个前向迭代器</span><br><span class="line"> listRewind(sentinel.scripts_queue,&li);</span><br><span class="line"> // 开始遍历running_scripts队列</span><br><span class="line"> while (sentinel.running_scripts < SENTINEL_SCRIPT_MAX_RUNNING &&</span><br><span class="line"> (ln = listNext(&li)) != NULL)</span><br><span class="line"> {</span><br><span class="line"> sentinelScriptJob *sj = ln->value;</span><br><span class="line"> pid_t pid;</span><br><span class="line"></span><br><span class="line"> /* Skip if already running. */</span><br><span class="line"> // 跳过正在执行的job</span><br><span class="line"> if (sj->flags & SENTINEL_SCRIPT_RUNNING) continue;</span><br><span class="line"></span><br><span class="line"> /* Skip if it's a retry, but not enough time has elapsed. */</span><br><span class="line"> // 还没到执行时间，暂时跳过</span><br><span class="line"> if (sj->start_time && sj->start_time > now) continue;</span><br><span class="line"></span><br><span class="line"> sj->flags |= SENTINEL_SCRIPT_RUNNING;</span><br><span class="line"> sj->start_time = mstime();</span><br><span class="line"> sj->retry_num++;</span><br><span class="line"> // fork一个子进程</span><br><span class="line"> pid = fork();</span><br><span class="line"></span><br><span class="line"> // fork子进程失败</span><br><span class="line"> if (pid == -1) {</span><br><span class="line"> /* Parent (fork error).</span><br><span class="line"> * We report fork errors as signal 99, in order to unify the</span><br><span class="line"> * reporting with other kind of errors. */</span><br><span class="line"> sentinelEvent(LL_WARNING,"-script-error",NULL,</span><br><span class="line"> "%s %d %d", sj->argv[0], 99, 0);</span><br><span class="line"> sj->flags &= ~SENTINEL_SCRIPT_RUNNING;</span><br><span class="line"> sj->pid = 0;</span><br><span class="line"> } else if (pid == 0) {</span><br><span class="line"> /* Child */</span><br><span class="line"> execve(sj->argv[0],sj->argv,environ);</span><br><span class="line"> /* If we are here an error occurred. */</span><br><span class="line"> _exit(2); /* Don't retry execution. */</span><br><span class="line"> } else {</span><br><span class="line"> sentinel.running_scripts++;</span><br><span class="line"> sj->pid = pid;</span><br><span class="line"> sentinelEvent(LL_DEBUG,"+script-child",NULL,"%ld",(long)pid);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">/* Check for scripts that terminated, and remove them from the queue if the</span><br><span class="line"> * script terminated successfully. If instead the script was terminated by</span><br><span class="line"> * a signal, or returned exit code "1", it is scheduled to run again if</span><br><span class="line"> * the max number of retries did not already elapsed. */</span><br><span class="line">void sentinelCollectTerminatedScripts(void) {</span><br><span class="line"> int statloc;</span><br><span class="line"> pid_t pid;</span><br><span class="line"></span><br><span class="line"> while ((pid = wait3(&statloc,WNOHANG,NULL)) > 0) {</span><br><span class="line"> int exitcode = WEXITSTATUS(statloc);</span><br><span class="line"> int bysignal = 0;</span><br><span class="line"> listNode *ln;</span><br><span class="line"> sentinelScriptJob *sj;</span><br><span class="line"></span><br><span class="line"> if (WIFSIGNALED(statloc)) bysignal = WTERMSIG(statloc);</span><br><span class="line"> sentinelEvent(LL_DEBUG,"-script-child",NULL,"%ld %d %d",</span><br><span class="line"> (long)pid, exitcode, bysignal);</span><br><span class="line"></span><br><span class="line"> ln = sentinelGetScriptListNodeByPid(pid);</span><br><span class="line"> if (ln == NULL) {</span><br><span class="line"> serverLog(LL_WARNING,"wait3() returned a pid (%ld) we can't find in our scripts execution queue!", (long)pid);</span><br><span class="line"> continue;</span><br><span class="line"> }</span><br><span class="line"> sj = ln->value;</span><br><span class="line"></span><br><span class="line"> /* If the script was terminated by a signal or returns an</span><br><span class="line"> * exit code of "1" (that means: please retry), we reschedule it</span><br><span class="line"> * if the max number of retries is not already reached. */</span><br><span class="line"> // 如果脚本中断或者退出值为1。则重新进入队列，并增加执行时间</span><br><span class="line"> if ((bysignal || exitcode == 1) &&</span><br><span class="line"> sj->retry_num != SENTINEL_SCRIPT_MAX_RETRY)</span><br><span class="line"> {</span><br><span class="line"> sj->flags &= ~SENTINEL_SCRIPT_RUNNING;</span><br><span class="line"> sj->pid = 0;</span><br><span class="line"> sj->start_time = mstime() +</span><br><span class="line"> sentinelScriptRetryDelay(sj->retry_num);</span><br><span class="line"> } else {</span><br><span class="line"> /* Otherwise let's remove the script, but log the event if the</span><br><span class="line"> * execution did not terminated in the best of the ways. */</span><br><span class="line"> // 如果是中断或者不成功，则是因为到达了执行次数上线，打印出错误日志</span><br><span class="line"> if (bysignal || exitcode != 0) {</span><br><span class="line"> sentinelEvent(LL_WARNING,"-script-error",NULL,</span><br><span class="line"> "%s %d %d", sj->argv[0], bysignal, exitcode);</span><br><span class="line"> }</span><br><span class="line"> // 这个地方只会在成功或者重试了10才执行到。</span><br><span class="line"> listDelNode(sentinel.scripts_queue,ln);</span><br><span class="line"> sentinelReleaseScriptJob(sj);</span><br><span class="line"> sentinel.running_scripts--;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="解决方案"><a href="#解决方案" class="headerlink" title="解决方案"></a>解决方案</h2><h3 id="临时解决方案"><a href="#临时解决方案" class="headerlink" title="临时解决方案"></a>临时解决方案</h3><p>脚本不exit 1。exit 2表示失败，即不进行哨兵重试调用脚本行为。</p><h3 id="源码修复"><a href="#源码修复" class="headerlink" title="源码修复"></a>源码修复</h3><p>将上面源码中的<code>sentinel.running_scripts--;</code>提到else之外。即使exit 1也需要减一。</p>]]></content>
<summary type="html">
<p>前一阵子一直在做自建机房Redis主从的环境搭建。了解到哨兵高可用切换后的会调用<code>client-reconfig-script</code>参数配置的脚本。<br>但是遇到了一个从2.8版本一直存在至今的BUG。我已经提了一个PR给官方，并被meger了。<a href="https://github.com/antirez/redis/pull/7113" target="_blank" rel="noopener">https://github.com/antirez/redis/pull/7113</a><br>特此记录一下。</p>
<h2 id="BUG场景"><a href="#BUG场景" class="headerlink" title="BUG场景"></a>BUG场景</h2><p>手动将主实例kill掉，模拟宕机情况。在某些情况下，<font color="red">哨兵已经触发了高可用切换</font>行为（主从状态、日志均有）。但是并<font color="red">没有调用</font>配置的脚本（非必现，但是落到同一台机器调用时并不会调用）<br>重启该机器上的哨兵节点又恢复正常。（重启大法好）</p>
</summary>
<category term="Redis" scheme="http://yoursite.com/categories/Redis/"/>
<category term="Redis" scheme="http://yoursite.com/tags/Redis/"/>
</entry>
</feed>