技术流ken

运维拯救世界

RabbitMQ消息队列–技术流ken

一、MQ 全称为 Message Queue, 消息队列( MQ )

是一种应用程序对应用程序的通信方法。应用程序通过读写出入队列的消息(针对应用程序的数据)来通信,而无需专用连接来链接它们。

消息传递指的是程序之间通过在消息中发送数据进行通信,而不是通过直接调用彼此来通信,直接调用通常是用于诸如远程过程调用的技术。排队指的是应用程序通过队列来通信。队列的使用除去了接收和发送应用程序同时执行的要求。

排队指的是应用程序通过 队列来通信。队列的使用除去了接收和发送应用程序同时执行的要求。

 

使用场景

在项目中,将一些无需即时返回且耗时的操作提取出来,进行了异步处理,而这种异步处理的方式大大的节省了服务器的请求响应时间,从而提高了系统的吞吐量。

 

二、AMQP  即 Advanced Message Queuing Protocol

高级消息队列协议,是应用层协议的一个开放标准,为面向消息的中间件设计。消息中间件主要用于组件之间的解耦,消息的发送者无需知道消息使用者的存在,反之亦然。

AMQP 的主要特征是面向消息、队列、路由(包括点对点和发布 / 订阅)、可靠性、安全。

三、 Rabbitmq概念:

属于一个流行的开源消息队列系统。属于AMQP( 高级消息队列协议 ) 标准的一个 实现。是应用层协议的一个开放标准,为面向消息的中间件设计。用于在分布式系统中存储转发消息,在 易用性、扩展性、高可用性等方面表现不俗。

消息中间件主要用于组件之间的解耦,消息的发送者无需知道消息使用者的存在,反之亦然。
AMQP 的主要特征是面向消息、队列、路由(包括点对点和发布 / 订阅)、可靠性、安全。

RabbitMQ特点:
使用Erlang编写
支持持久化
支持HA
提供C# , erlang,java,perl,python,ruby等的client开发端

四、什么是耦合、解耦合

一、耦合
1、耦合是指两个或两个以上的体系或两种运动形式间通过相互作用而彼此影响以至联合起来的现象。

2、在软件工程中,对象之间的耦合度就是对象之间的依赖性。对象之间的耦合越高,维护成本越高,因此对象的设计应使类和构件之间的耦合最小。

3、分类:有软硬件之间的耦合,还有软件各模块之间的耦合。耦合性是程序结构中各个模块之间相互关联的度量。它取决于各个模块之间的接口的复杂程度、调用模块的方式以及哪些信息通过接口。

二、解耦
1、解耦,字面意思就是解除耦合关系。

2、在软件工程中,降低耦合度即可以理解为解耦,模块间有依赖关系必然存在耦合,理论上的绝对零耦合是做不到的,但可以通过一些现有的方法将耦合度降至最低。

3、设计的核心思想:尽可能减少代码耦合,如果发现代码耦合,就要采取解耦技术。让数据模型,业务逻辑和视图显示三层之间彼此降低耦合,把关联依赖降到最低,而不至于牵一发而动全身。原则就是A功能的代码不要写在B的功能代码中,如果两者之间需要交互,可以通过接口,通过消息,甚至可以引入框架,但总之就是不要直接交叉写。

五、RabbitMQ中的概念名词

Broker:简单来说就是消息队列服务器实体。

Exchange:消息交换机,它指定消息按什么规则,路由到哪个队列。

Queue:消息队列载体,每个消息都会被投入到一个或多个队列。

Binding:绑定,它的作用就是把exchange和queue按照路由规则绑定起来。

Routing Key:路由关键字, exchange根据这个关键字进行消息投递。

vhost:虚拟主机,一个broker里可以开设多个vhost,用作不同用户的权限分离。

producer:消息生产者,就是投递消息的程序。

consumer:消息消费者,就是接受消息的程序。

channel:消息通道,在客户端的每个连接里,可建立多个channel,每个 channel代表一个会话任务。

六、RabbitMQ工作原理

 

MQ 是消费 – 生产者模型的一个典型的代表,一端往消息队列中不断写入消息,而另一端则可以读取或者订阅队列中的消息。 MQ 则是遵循了 AMQP协议的具体实现和产品。在项目中,将一些无需即时返回且耗时的操作提取出来,进行了异步处理,而这种异步处理的方式大大的节省了服务器的请求响应时间,从而提高了系统的吞吐量。

( 1)客户端连接到消息队列服务器,打开一个channel。

( 2)客户端声明一个exchange,并设置相关属性。

( 3)客户端声明一个queue,并设置相关属性。

( 4)客户端使用routing key,在exchange和queue之间建立好绑定关系。

( 5)客户端投递消息到exchange。

( 6) exchange接收到消息后,就根据消息的key和已经设置的binding,进 行消息路由,将消息投递到一个或多个队列里

 

 

exchange也有几个类型,完全根据key进行投递的叫做Direct交换机,例如,绑定时设置了routing key为”abc”,那么客户端提交的消息,只有设置了key为”abc”的才会投递到队列。对key进行模式匹配后进行投递的叫做Topic交换机,符号”#”匹配一个或多个词,符号”*”匹配正好一个词。例如”abc.#”匹配”abc.def.ghi”,”abc.*”只匹配”abc.def”。还有一种不需要key的,叫做Fanout交换机,它采取广播模式,一个消息进来时,投递到与该交换机绑定的所有队列。

七、Rabbitmq 的 metadata

元数据可以持久化在 RAM 或 Disc. 从这个角度可以把 RabbitMQ 集群中的节点分成两种 :RAM Node和 Disk Node.

RAM Node 只会将元数据存放在RAM

Disk node 会将元数据持久化到磁盘。

单节点系统就没有什么选择了 , 只允许 disk node, 否则由于没有数据冗余一旦重启就会丢掉所有的配置信息 . 但在集群环境中可以选择哪些节点是 RAM node.在集群中声明(declare) 创建 exchange queue binding, 这类操作要等到所有的节点都完成创建才会返回 :
如果是内存节点就要修改内存数据 ,
如果是 disk node 就要等待写磁盘 , 节点过多这里的速度就会被大大的拖慢 .

有些场景 exchang queue 相当固定 , 变动很少 ,那即使全都是 disc node, 也没有什么影响 . 如果使用 Rabbitmq 做 RPC( RPC :Remote Procedure Call—远程过程调用),  RPC 或者类似 RPC 的场景这个问题就严重了 , 频繁创建销毁临时队列 , 磁盘读写能力就很快成为性能瓶颈了。所以 , 大多数情况下 , 我们尽量把 Node 创建为RAM Node. 这里就有一个问题了 , 要想集群重启后元数据可以恢复就需要把集群元数据持久化到磁盘 , 那需要规划 RabbitMQ 集群中的 RAM Node 和 Disc Node 。

只要有一个节点是 Disc Node 就能提供条件把集群元数据写到磁盘 ,RabbitMQ 的确也是这样要求的 : 集群中只要有一个 disk node 就可以 , 其它的都可以是 RAM node. 节点加入或退出集群一定至少要通知集群中的一个 disk node 。

如果集群中 disk node 都宕掉 , 就不要变动集群的元数据 . 声明 exchange queue 修改用户权限 , 添加用户等等这些变动在节点重启之后无法恢复 。

有一种情况要求所有的 disk node 都要在线情况在才能操作 , 那就是增加或者移除节点 .RAM node 启动的时候会连接到预设的 disk node 下载最新的集群元数据 . 如果你有两个 disk node(d1 d2), 一个 RAM node 加入的时候你只告诉 d1, 而恰好这个 RAM node 重启的时候 d1 并没有启动 , 重启就会失败 . 所以加入 RAM 节点的时候 , 把所有的disk node 信息都告诉它 ,RAM node 会把 disk node 的信息持久化到磁盘以便后续启动可以按图索骥 .

八、Rabbitmq 集群部署

 

一、前期准备
(1)条件:准备3台linux系统,确保配置好源,及epel源
(2)三台机器能够静态解析彼此
(3)设置可以无密钥登陆
二、安装过程:
(1)所有node安装rabbtimq和erlang软件包:
yum install -y erlang rabbitmq-server.noarch
systemctl enable rabbitmq-server.service
systemctl start rabbitmq-server.service
systemctl status rabbitmq-server.service
查看监听端口:
netstat -lantp | grep 5672
配置文件:
vim /etc/rabbitmq/rabbitmq.config
(2)node1:修改guest密码为admin(默认用户为:guest 密码为:guest)
rabbitmqctl change_password guest admin
(3)node1:添加一个openstack的用户,并设密码为admin。并设置权限和成为管理员
node1:
rabbitmqctl add_user openstack admin
rabbitmqctl set_permissions openstack “.*” “.*” “.*”
rabbitmqctl set_user_tags openstack administrator
(4)node1:编辑rabbittmq变量文件
vim /etc/rabbitmq/rabbitmq-env.conf
RABBITMQ_NODE_PORT=5672
ulimit -S -n 4096
RABBITMQ_SERVER_ERL_ARGS=”+K true +A30 +P 1048576 -kernel inet_default_connect_options [{nodelay,true},{raw,6,18,<<5000:64/native>>}] -kernel inet_default_listen_options [{raw,6,18,<<5000:64/native>>}]”
RABBITMQ_NODE_IP_ADDRESS=172.16.254.60
(5)node1:将rabbittmq变量文件拷贝到其他两节点,之后并修改相应节点的ip
scp /etc/rabbitmq/rabbitmq-env.conf con2:/etc/rabbitmq/
scp /etc/rabbitmq/rabbitmq-env.conf con3:/etc/rabbitmq/
查看rabbitmq插件
/usr/lib/rabbitmq/bin/rabbitmq-plugins list
(6)所有node 开启rabbitmq的web管理页面
/usr/lib/rabbitmq/bin/rabbitmq-plugins enable rabbitmq_management mochiweb webmachine rabbitmq_web_dispatch amqp_client rabbitmq_management_agent
或者:rabbitmq-plugins enable rabbitmq_management
systemctl restart rabbitmq-server.service
systemctl status rabbitmq-server.service
(7)node1发送erlang.cookie到其他节点配置集群
rabbitmqctl status
scp /var/lib/rabbitmq/.erlang.cookie con2:/var/lib/rabbitmq/.erlang.cookie
scp /var/lib/rabbitmq/.erlang.cookie con3:/var/lib/rabbitmq/.erlang.cookie
(8)node2和node3停止应用,并以ram的方式加入node1节点,之后重启应用
systemctl restart rabbitmq-server.service
rabbitmqctl stop_app
rabbitmqctl join_cluster –ram rabbit@con1
rabbitmqctl start_app
(9)node1检查集群状态
[root@con1 conf]# rabbitmqctl cluster_status
Cluster status of node rabbit@con1 …
[{nodes,[{disc,[rabbit@con1]},{ram,[rabbit@con3,rabbit@con2]}]},
{running_nodes,[rabbit@con3,rabbit@con2,rabbit@con1]},
{cluster_name,<<“rabbit@con1”>>},
{partitions,[]},
{alarms,[{rabbit@con3,[]},{rabbit@con2,[]},{rabbit@con1,[]}]}]
(10)登陆验证:http://172.16.254.60:15672/#/             guest/admin
其他命令:
(1)添加管理员:
rabbitmqctl add_user mqadmin mqadmin
rabbitmqctl set_user_tags mqadmin administrator
rabbitmqctl set_permissions -p / mqadmin “.*” “.*” “.*”
(2)更改节点类型(内存型或磁盘型)
rabbitmqctl stop_app
rabbitmqctl change_cluster_node_type disc 或 rabbitmqctl change_cluster_node_type ram
rabbitmqctl start_app
(3)从集群移除节点(或者重置节点)
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
rabbitmqctl cluster_status
(4)从某个节点移除集群中其他节点
rabbitmqctl forget_cluster_node rabbit@node3
rabbitmqctl reset
rabbitmqctl start_app
rabbitmqctl cluster_status
1. 保证集群中至少有一个磁盘类型的节点以防数据丢失,在更改节点类型时尤其要注意。
2. 若整个集群被停掉了,应保证最后一个 down 掉的节点被最先启动,若不能则要使用 forget_cluster_node 命令将其移出集群
3. 若集群中节点几乎同时以不可控的方式 down 了此时在其中一个节点使用 force_boot 命令重启节点

9 thoughts on “RabbitMQ消息队列–技术流ken

  1. This is the correct weblog for anyone who wants to seek out out about this topic. You notice a lot its almost arduous to argue with you (not that I truly would want匟aHa). You undoubtedly put a brand new spin on a subject thats been written about for years. Nice stuff, simply nice!

  2. I simply desired to thank you so much once more. I am not sure the things that I could possibly have achieved in the absence of those hints shared by you relating to such a subject. Previously it was a hard condition in my view, however , taking note of your expert form you processed that forced me to cry over happiness. I’m grateful for the service and as well , wish you realize what a powerful job you’re undertaking educating the others with the aid of your web blog. I am certain you haven’t encountered all of us.

  3. My spouse and i ended up being now thankful that Chris managed to finish off his inquiry through your precious recommendations he acquired in your web site. It is now and again perplexing to simply happen to be offering guidance which often people today may have been selling. And we remember we have got the writer to thank because of that. The entire explanations you have made, the straightforward website menu, the relationships you assist to foster – it’s most great, and it’s helping our son and our family understand the topic is excellent, which is seriously essential. Thanks for all the pieces!

  4. I wish to point out my appreciation for your generosity supporting men and women that require assistance with this question. Your real commitment to passing the solution all over became wonderfully significant and has specifically made guys like me to arrive at their objectives. Your new important guidelines means so much to me and additionally to my peers. Warm regards; from all of us.

  5. I intended to draft you a very little word to say thanks a lot as before for the spectacular suggestions you have featured in this article. This has been certainly strangely open-handed of people like you to provide freely what many individuals could have offered for sale for an ebook to generate some bucks for themselves, primarily given that you could possibly have done it in the event you considered necessary. The solutions additionally served to provide a fantastic way to fully grasp other people online have the same eagerness much like my personal own to understand great deal more on the topic of this issue. I am sure there are millions of more pleasurable instances ahead for many who browse through your blog post.

  6. I precisely wished to appreciate you once more. I do not know what I would’ve carried out without the methods shared by you on such a area of interest. It was actually a real challenging case in my circumstances, nevertheless coming across this specialized way you dealt with that made me to jump with joy. I will be thankful for the work and in addition have high hopes you are aware of a great job your are getting into training most people using a blog. I am certain you haven’t got to know all of us.

  7. I’m commenting to let you understand what a great experience my cousin’s child had checking your web site. She even learned numerous issues, including what it is like to have a marvelous teaching nature to make the others completely gain knowledge of several tricky subject areas. You undoubtedly did more than our desires. Thank you for imparting the valuable, healthy, revealing not to mention cool tips on that topic to Kate.

  8. I precisely desired to appreciate you once again. I’m not certain the things I might have taken care of in the absence of the entire points shared by you concerning such theme. It truly was a real frustrating problem in my view, but observing a new professional manner you managed that took me to jump over gladness. I’m happy for your advice and as well , have high hopes you know what a great job you are always doing teaching other individuals through your site. I am certain you have never encountered all of us.

  9. My husband and i got fulfilled Jordan managed to do his analysis from your ideas he made in your site. It’s not at all simplistic just to always be giving for free secrets which usually a number of people might have been trying to sell. We know we have got you to be grateful to because of that. All the illustrations you made, the straightforward website navigation, the friendships your site make it easier to create – it’s all astonishing, and it’s making our son and the family believe that this article is excellent, which is certainly unbelievably vital. Many thanks for all!

发表评论

电子邮件地址不会被公开。