May 7

[原]解决 OpenvSwitch terminating with signal 14 (Alarm clock) 错误 雨

linuxing , 14:33 , 网络服务 » OpenStack , 评论(0) , 引用(0) , 阅读(39380) , Via 本站原创 | |
在配置br-ex 桥接到eth0 网卡后,重启neutron-openvswitch-agent 服务后,一直提示报错,无法创建patch-int 和 patch-tun 网卡(极少时候是可以的)。
这导致openvswitch 在不断的重启,而对外的网络(与RabbitMQ 连接)也在不断的重启中。

日志:
引用
2015-05-06 23:40:43.299 18254 ERROR neutron.agent.linux.ovs_lib [req-9ec5d95e-3626-4494-b043-35d5211747d8 None] Unable to execute ['ovs-vsctl', '--timeout=10', 'add-port', 'br-int', 'patch-tun', '--', 'set', 'Interface', 'patch-tun', 'type=patch', 'options:peer=patch-int']. Exception:
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', 'add-port', 'br-int', 'patch-tun', '--', 'set', 'Interface', 'patch-tun', 'type=patch', 'options:peer=patch-int']
Exit code: 242
Stdout: ''
Stderr: '2015-05-06T15:40:43Z|00002|fatal_signal|WARN|terminating with signal 14 (Alarm clock)\n'
2015-05-06 23:40:45.936 18254 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 192.168.209.137:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 9 seconds.
2015-05-06 23:40:53.530 18254 ERROR neutron.agent.linux.utils [req-9ec5d95e-3626-4494-b043-35d5211747d8 None]
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', 'add-port', 'br-tun', 'patch-int', '--', 'set', 'Interface', 'patch-int', 'type=patch', 'options:peer=patch-tun']
Exit code: 242
Stdout: ''
Stderr: '2015-05-06T15:40:53Z|00002|fatal_signal|WARN|terminating with signal 14 (Alarm clock)\n'2015-05-06 23:40:53.530 18254 ERROR neutron.agent.linux.ovs_lib [req-9ec5d95e-3626-4494-b043-35d5211747d8 None] Unable to execute ['ovs-vsctl', '--timeout=10', 'add-port', 'br-tun', 'patch-int', '--', 'set', 'Interface', 'patch-int', 'type=patch', 'options:peer=patch-tun']. Exception:
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', 'add-port', 'br-tun', 'patch-int', '--', 'set', 'Interface', 'patch-int', 'type=patch', 'options:peer=patch-tun']
Exit code: 242
Stdout: ''
Stderr: '2015-05-06T15:40:53Z|00002|fatal_signal|WARN|terminating with signal 14 (Alarm clock)\n'
2015-05-06 23:40:53.638 18254 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-9ec5d95e-3626-4494-b043-35d5211747d8 None] Failed to create OVS patch port. Cannot have tunneling enabled on this agent, since this version of OVS does not support tunnels or patch ports. Agent terminated!2015-05-06 23:48:13.179 18701 INFO neutron.common.config [-] Logging enabled!
2015-05-06 23:48:13.728 18701 ERROR neutron.agent.linux.utils [-]

1.排查
经排查,原因是Juno 版 neutron-openvswitch-agent 在计算节点环境下,重启时有部分遗留的mask 没有清除,导致加入patch-int 等port 的时候,OpenvSwitch 进程会出现异常。

在网络节点上采用相同的openvswitch 和 neutron-openvswitch-agent 与配置是正常的。

相关Bug:
Bug 1185521 - ovs-vswitchd crash [NEEDINFO]
https://bugzilla.redhat.com/show_bug.cgi?id=1185521

2.解决
升级到 openvswitch-2.1.2-2.el7.centos.2.x86_64.rpm 即可。
引用
[root@compute01 ~]# rpm -qp openvswitch-2.1.2-2.el7_0.2.src.rpm --changelog|more
* Thu Feb 12 2015 Flavio Leitner - 2.1.2-2.el7_0.2
- applied fix to not use mask if it doesn't exist (#1187257)

升级版本:
引用
[root@compute01 ~]# rpm -Uvh /root/rpmbuild/RPMS/x86_64/openvswitch-2.1.2-2.el7.centos.2.x86_64.rpm /root/rpmbuild/RPMS/noarch/python-openvswitch-2.1.2-2.el7.centos.2.noarch.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:python-openvswitch-2.1.2-2.el7.ce################################# [ 33%]
   2:openvswitch-2.1.2-2.el7.centos.2 ################################# [ 67%]
Cleaning up / removing...
   3:openvswitch-2.1.2-2.el7.centos.1 ################################# [100%]

[root@compute01 ~]# rpm -qa|grep openvswitch
python-openvswitch-2.1.2-2.el7.centos.2.noarch
openstack-neutron-openvswitch-2014.2.2-1.el7.noarch
openvswitch-2.1.2-2.el7.centos.2.x86_64

重新配置br-ex 桥接:
引用
[root@compute01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-br-ex
DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=192.168.209.138
NETMASK=255.255.255.0
GATEWAY=192.168.209.1
DNS1=10.199.129.21
ONBOOT=yes
[root@compute01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
DEVICETYPE=ovs
TYPE=OVSPort
OVS_BRIDGE=br-ex
ONBOOT=yes
BOOTPROTO=none
[root@compute01 ~]# systemctl restart network
[root@compute01 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.209.1   0.0.0.0         UG    0      0        0 br-ex
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     1075   0        0 br-ex
172.16.130.0    0.0.0.0         255.255.255.0   U     0      0        0 eth1
192.168.209.0   0.0.0.0         255.255.255.0   U     0      0        0 br-ex
[root@compute01 ~]# ovs-vsctl show
e65c1559-873e-41a5-9ff9-c2831d17bb95
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-tun
        Port "vxlan-ac10828b"
            Interface "vxlan-ac10828b"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.130.138", out_key=flow, remote_ip="172.16.130.139"}
        Port "vxlan-ac108289"
            Interface "vxlan-ac108289"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.16.130.138", out_key=flow, remote_ip="172.16.130.137"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    Bridge br-ex
        Port "eth0"
            Interface "eth0"
        Port br-ex
            Interface br-ex
                type: internal
    ovs_version: "2.1.3"
[root@compute01 ~]# systemctl restart neutron-openvswitch-agent.service

[root@compute01 ~]# tail -f /var/log/messages
May  7 13:26:33 compute01 systemd: Stopping OpenStack Neutron Open vSwitch Agent...
May  7 13:26:35 compute01 systemd: Starting OpenStack Neutron Open vSwitch Agent...
May  7 13:26:35 compute01 systemd: Started OpenStack Neutron Open vSwitch Agent.
May  7 13:26:36 compute01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 -- --may-exist add-br br-int
May  7 13:26:36 compute01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 -- set-fail-mode br-int secure
May  7 13:26:36 compute01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 -- --if-exists del-port br-int patch-tun
May  7 13:26:37 compute01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 -- --if-exists del-br br-tun
May  7 13:26:37 compute01 kernel: device br-tun left promiscuous mode
May  7 13:26:37 compute01 avahi-daemon[614]: Withdrawing workstation service for br-tun.
May  7 13:26:37 compute01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 -- --may-exist add-br br-tun
May  7 13:26:37 compute01 kernel: device br-tun entered promiscuous mode
May  7 13:26:37 compute01 systemd-sysctl: Overwriting earlier assignment of net/bridge/bridge-nf-call-ip6tables in file '/etc/sysctl.d/99-sysctl.conf'.
May  7 13:26:37 compute01 systemd-sysctl: Overwriting earlier assignment of net/bridge/bridge-nf-call-iptables in file '/etc/sysctl.d/99-sysctl.conf'.
May  7 13:26:37 compute01 systemd-sysctl: Overwriting earlier assignment of net/bridge/bridge-nf-call-arptables in file '/etc/sysctl.d/99-sysctl.conf'.
May  7 13:26:37 compute01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 add-port br-int patch-tun -- set Interface patch-tun type=patch options:peer=patch-int
May  7 13:26:37 compute01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 add-port br-tun patch-int -- set Interface patch-int type=patch options:peer=patch-tun
May  7 13:26:39 compute01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 -- --may-exist add-port br-tun vxlan-ac108289 -- set Interface vxlan-ac108289 type=vxlan options:df_default=true options:remote_ip=172.16.130.137 options:local_ip=172.16.130.138 options:in_key=flow options:out_key=flow
May  7 13:26:40 compute01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /bin/ovs-vsctl --timeout=10 -- --may-exist add-port br-tun vxlan-ac10828b -- set Interface vxlan-ac10828b type=vxlan options:df_default=true options:remote_ip=172.16.130.139 options:local_ip=172.16.130.138 options:in_key=flow options:out_key=flow

[root@compute01 ~]# tail -f /var/log/neutron/openvswitch-agent.log
2015-05-07 13:26:36.291 17432 INFO neutron.common.config [-] Logging enabled!
2015-05-07 13:26:37.078 17432 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 192.168.209.137:5672
2015-05-07 13:26:37.120 17432 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 192.168.209.137:5672
2015-05-07 13:26:37.126 17432 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 192.168.209.137:5672
2015-05-07 13:26:37.148 17432 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 192.168.209.137:5672
2015-05-07 13:26:39.403 17432 INFO oslo.messaging._drivers.impl_rabbit [req-e4e1fce6-9c83-4668-a0c7-9855981bd807 ] Connecting to AMQP server on 192.168.209.137:5672
2015-05-07 13:26:39.425 17432 INFO oslo.messaging._drivers.impl_rabbit [req-e4e1fce6-9c83-4668-a0c7-9855981bd807 ] Connected to AMQP server on 192.168.209.137:5672
2015-05-07 13:26:39.438 17432 INFO oslo.messaging._drivers.impl_rabbit [req-e4e1fce6-9c83-4668-a0c7-9855981bd807 ] Connecting to AMQP server on 192.168.209.137:5672
2015-05-07 13:26:39.460 17432 INFO oslo.messaging._drivers.impl_rabbit [req-e4e1fce6-9c83-4668-a0c7-9855981bd807 ] Connected to AMQP server on 192.168.209.137:5672
2015-05-07 13:26:39.470 17432 INFO oslo.messaging._drivers.impl_rabbit [req-e4e1fce6-9c83-4668-a0c7-9855981bd807 ] Connecting to AMQP server on 192.168.209.137:5672
2015-05-07 13:26:39.491 17432 INFO oslo.messaging._drivers.impl_rabbit [req-e4e1fce6-9c83-4668-a0c7-9855981bd807 ] Connected to AMQP server on 192.168.209.137:5672
2015-05-07 13:26:39.501 17432 INFO oslo.messaging._drivers.impl_rabbit [req-e4e1fce6-9c83-4668-a0c7-9855981bd807 ] Connecting to AMQP server on 192.168.209.137:5672
2015-05-07 13:26:39.522 17432 INFO oslo.messaging._drivers.impl_rabbit [req-e4e1fce6-9c83-4668-a0c7-9855981bd807 ] Connected to AMQP server on 192.168.209.137:5672
2015-05-07 13:26:39.534 17432 INFO oslo.messaging._drivers.impl_rabbit [req-e4e1fce6-9c83-4668-a0c7-9855981bd807 ] Connecting to AMQP server on 192.168.209.137:5672
2015-05-07 13:26:39.558 17432 INFO oslo.messaging._drivers.impl_rabbit [req-e4e1fce6-9c83-4668-a0c7-9855981bd807 ] Connected to AMQP server on 192.168.209.137:5672
2015-05-07 13:26:39.570 17432 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-e4e1fce6-9c83-4668-a0c7-9855981bd807 None] Agent initialized successfully, now running...
2015-05-07 13:26:39.581 17432 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-e4e1fce6-9c83-4668-a0c7-9855981bd807 None] Agent out of sync with plugin!
2015-05-07 13:26:39.719 17432 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-e4e1fce6-9c83-4668-a0c7-9855981bd807 None] Agent tunnel out of sync with plugin!

桥接br-ex 网卡以及创建path-tun 等接口正常,问题解决。
发表评论
表情
emotemotemotemotemot
emotemotemotemotemot
emotemotemotemotemot
emotemotemotemotemot
emotemotemotemotemot
打开HTML
打开UBB
打开表情
隐藏
记住我
昵称   密码   游客无需密码
网址   电邮   [注册]