{"id":1137,"date":"2017-09-05T23:00:00","date_gmt":"2017-09-05T23:00:00","guid":{"rendered":"https:\/\/fir3netwp.gmsrrpobkbd.com\/2017\/09\/05\/troubleshooting-connectivity-to-the-neutron-metadata-proxy\/"},"modified":"2021-07-31T15:41:09","modified_gmt":"2021-07-31T15:41:09","slug":"troubleshooting-connectivity-to-the-neutron-metadata-proxy","status":"publish","type":"post","link":"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html","title":{"rendered":"Troubleshooting Connectivity to the Neutron Metadata Proxy"},"content":{"rendered":"<p>&nbsp;<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_65 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-662e796aa544a\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-662e796aa544a\"  aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\/#Introduction\" title=\"Introduction\">Introduction<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\/#Issue\" title=\"Issue\">Issue<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\/#Troubleshooting\" title=\"Troubleshooting\">Troubleshooting<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\/#Guest_%E2%80%93_Injected_Route\" title=\"Guest &#8211; Injected Route\">Guest &#8211; Injected Route<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\/#Compute_%E2%80%93_Bridge_Packet_Trace\" title=\"Compute &#8211; Bridge Packet Trace\">Compute &#8211; Bridge Packet Trace<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\/#Confirm_Instance_Name\" title=\"Confirm Instance Name\">Confirm Instance Name<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\/#Confirm_Bridge_Name\" title=\"Confirm Bridge Name\">Confirm Bridge Name<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\/#TCPDump\" title=\"TCPDump\">TCPDump<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\/#Controller_%E2%80%93_Neutron_DHCP_Namespace\" title=\"Controller &#8211; Neutron DHCP Namespace\">Controller &#8211; Neutron DHCP Namespace<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\/#Validation\" title=\"Validation\">Validation<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\/#Summary\" title=\"Summary\">Summary<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\/#References\" title=\"References\">References<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"Introduction\"><\/span>Introduction<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p class=\"c0\"><span class=\"c1\">After troubleshooting a recent issue with accessing meta services from a guest instance, and jumping through the various steps within the path, I soon realised &#8211; this would make for a great article.<\/span><\/p>\n<h2 class=\"c0\"><span class=\"ez-toc-section\" id=\"Issue\"><\/span><span class=\"c1\">Issue<\/span><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p class=\"c0\"><span class=\"c1\">So first of all let&#8217;s look at the issue. The issue was pretty basic. Quite simply I was unable to connect to the metadata service from a nova instance. As shown below:<\/span><\/p>\n<pre class=\"prettyprint\">$ curl -v  \r\n* couldn't connect to host\r\ncurl: (7) couldn't connect to host<\/pre>\n<h2 class=\"c0\"><span class=\"ez-toc-section\" id=\"Troubleshooting\"><\/span><span class=\"c1\">Troubleshooting<\/span><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p class=\"c0\"><span class=\"c1\">In order to troubleshoot we will step through each part of the path &#8211; from guest (upon the compute node) to the Neutron metadata proxy (within the infra nodes and relating LXC) .<\/span><\/p>\n<p class=\"c0\"><span class=\"c1\">Please note that the OpenStack environment within this article is based upon a Linux Bridge OSA (OpenStack Ansible) based deployment.<\/span><\/p>\n<h2 class=\"c0\"><span class=\"ez-toc-section\" id=\"Guest_%E2%80%93_Injected_Route\"><\/span><span class=\"c1\">Guest &#8211; Injected Route<\/span><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p class=\"c0\">As our guest is on an isolated network, i.e a network that is not behind an L3 router. In addition our DHCP agent is configured with <span class=\"codeinline\">enable_isolated_metadata = True<\/span><sup class=\"c4\"><a id=\"ftnt_ref1\" href=\"#ftnt1\">[1]<\/a><\/sup><span class=\"c4\">, \u00a0meaning our DHCP server will append specific host routes to the DHCP request (via option 121). This results in an additional route being added to the guest, in turn ensuring requests to <\/span>169.254.169.254 are routed via the DHCP agent IP.<\/p>\n<pre class=\"prettyprint\">$ netstat -nr\r\nKernel IP routing table\r\nDestination     Gateway         Genmask         Flags   MSS Window  irtt Iface\r\n0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth0\r\n169.254.169.254 192.168.1.2     255.255.255.255 UGH       0 0          0 eth0\r\n192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0<\/pre>\n<p class=\"c0\"><span class=\"c3\">As you can see the correct route is in place. Lets ping the address to confirm connectivity:<\/span><\/p>\n<pre class=\"prettyprint\">$ ping 192.168.1.2 -c 2\r\nPING 192.168.1.2 (192.168.1.2): 56 data bytes\r\n64 bytes from 192.168.1.2: seq=0 ttl=64 time=0.768 ms\r\n64 bytes from 192.168.1.2: seq=1 ttl=64 time=0.545 ms\r\nThis all looks good, lets move on.<\/pre>\n<p class=\"c0 c2\">This all looks good, lets move on.<\/p>\n<h2 class=\"c0\"><span class=\"ez-toc-section\" id=\"Compute_%E2%80%93_Bridge_Packet_Trace\"><\/span><span class=\"c1\">Compute &#8211; Bridge Packet Trace<\/span><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p class=\"c0\"><span class=\"c1\">Next we will confirm that traffic is making it out from the instance to the bridge.\u00a0<\/span><span class=\"c1\">There are a few steps in this process. They are:<\/span><\/p>\n<h3 class=\"c0\"><span class=\"ez-toc-section\" id=\"Confirm_Instance_Name\"><\/span><span class=\"c1\">Confirm Instance Name<\/span><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p class=\"c0\"><span class=\"c1\">In order to obtain the instance name, we perform a show on the server id. Like so,<\/span><\/p>\n<pre class=\"prettyprint\">root@infra1:~# openstack server list\r\n+--------------------------------------+-------------------------+--------+------------------------------------------------------------+------------------------+\r\n| ID                                   | Name                    | Status | Networks                                                   | Image Name             |\r\n+--------------------------------------+-------------------------+--------+------------------------------------------------------------+------------------------+                                                   \r\n| d0cebae2-ba1a-45fc-8f11-d28e8fd93ee3 | cirros-DEMO             | ACTIVE | tenant_network_lab001-vsrx=192.168.1.50                    | cirros                 |\r\n+--------------------------------------+-------------------------+--------+------------------------------------------------------------+------------------------+\r\n\r\nroot@infra1:~# openstack server show d0cebae2-ba1a-45fc-8f11-d28e8fd93ee3\r\n+-------------------------------------+----------------------------------------------------------+\r\n| Field                               | Value                                                    |\r\n+-------------------------------------+----------------------------------------------------------+\r\n| OS-DCF:diskConfig                   | MANUAL                                                   |\r\n| OS-EXT-AZ:availability_zone         | nova                                                     |\r\n| OS-EXT-SRV-ATTR:host                | compute03                                                |\r\n| OS-EXT-SRV-ATTR:hypervisor_hostname | compute03                                                |\r\n| OS-EXT-SRV-ATTR:instance_name       | instance-0000001a                                        |\r\n| OS-EXT-STS:power_state              | Running                                                  |\r\n| OS-EXT-STS:task_state               | None                                                     |\r\n| OS-EXT-STS:vm_state                 | active                                                   |\r\n| OS-SRV-USG:launched_at              | 2017-09-03T06:19:23.000000                               |\r\n| OS-SRV-USG:terminated_at            | None                                                     |\r\n| accessIPv4                          |                                                          |\r\n| accessIPv6                          |                                                          |\r\n| addresses                           | tenant_network_lab001-vsrx=192.168.1.50                  |\r\n| config_drive                        |                                                          |\r\n| created                             | 2017-09-03T06:19:18Z                                     |\r\n| flavor                              | cirros-small (58772350-942d-442b-9e39-ef953787a2d4)      |\r\n| hostId                              | 57f31ea4cecf1bdae04624374a6db8df774b57e94d8427a8c5f1cff1 |\r\n| id                                  | d0cebae2-ba1a-45fc-8f11-d28e8fd93ee3                     |\r\n| image                               | cirros (5ffaa767-fc80-4d92-90f7-29b6d949797f)            |\r\n| key_name                            | None                                                     |\r\n| name                                | cirros-DEMO                                              |\r\n| progress                            | 0                                                        |\r\n| project_id                          | f1eb80264e9c4c688c7603bbb5541396                         |\r\n| properties                          |                                                          |\r\n| status                              | ACTIVE                                                   |\r\n| updated                             | 2017-09-03T06:19:23Z                                     |\r\n| user_id                             | 85aea2625c7349fc8796527085a04226                         |\r\n| volumes_attached                    |                                                          |\r\n+-------------------------------------+----------------------------------------------------------+<\/pre>\n<p class=\"c0\"><span class=\"c1\">\u00a0From the output can you see the\u00a0<\/span><span class=\"c1\">field <span class=\"codeinline\">OS-EXT-SRV-ATTR:instance_name<\/span> and its value &#8211; <span class=\"codeinline\">instance-0000001a<\/span>.<\/span><\/p>\n<h3 class=\"c0\"><span class=\"ez-toc-section\" id=\"Confirm_Bridge_Name\"><\/span><span class=\"c1\">Confirm Bridge Name<\/span><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p class=\"c0\"><span class=\"c1\">Using the <span class=\"codeinline\">instance_name<\/span> we can then perform a lookup (on the corresponding host) for the bridge. As shown below,<\/span><\/p>\n<pre class=\"prettyprint\">root@compute03:~# virsh domiflist instance-0000001a\r\nInterface      Type   Source         Model   MAC\r\n--------------------------------------------------------------\r\ntap0e6c063c-48 bridge brq423b2b1b-55 virtio  fa:16:3e:92:bd:fd<\/pre>\n<h3 class=\"c0\"><span class=\"ez-toc-section\" id=\"TCPDump\"><\/span>TCPDump<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p class=\"c0\"><span class=\"c1\">Now we have the bridge name, whilst running another curl from the guest\u00a0we can perform a tcpdump. From this we can see the SYN is sent, but a RST is returned. So we know the traffic from the guest is making it to the attached bridge, however something is preventing the completion of the TCP connection by sending a RST.\u00a0<\/span><\/p>\n<pre class=\"printprint\">root@compute03:~# tcpdump -ni brq423b2b1b-55\r\ntcpdump: verbose output suppressed, use -v or -vv for full protocol decode\r\nlistening on brq423b2b1b-55, link-type EN10MB (Ethernet), capture size 262144 bytes\r\n22:02:27.792813 IP 192.168.1.50.49513 &gt; 169.254.169.254.80: Flags [S], seq 2770101210, win 14100, options [mss 1410,sackOK,TS val 77970740 ecr 0,nop,wscale 2], length 0\r\n22:02:27.793369 IP 169.254.169.254.80 &gt; 192.168.1.50.49513: Flags [R.], seq 0, ack 2770101211, win 0, length 0<\/pre>\n<h2 class=\"c0\"><span class=\"ez-toc-section\" id=\"Controller_%E2%80%93_Neutron_DHCP_Namespace\"><\/span><span class=\"c1\">Controller &#8211; Neutron DHCP Namespace<\/span><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p class=\"c0\"><span class=\"c1\">We will now jump into the DHCP namespace. This namespace also hosts the metadata service proxy. \u00a0Once inside we can confirm if the necessary IPs are configured, and also run another tcpdump to see if the traffic is making it in.<\/span><\/p>\n<p class=\"c0\"><span class=\"c1\">First things first, upon the infra node we connect to the <span class=\"codeinline\">neutron_agents_container<\/span>. We confirm its full name by running <span class=\"codeinline\">lxc-ls -f<\/span>.<\/span><\/p>\n<pre class=\"prettyprint\"><span class=\"c1\">lxc-attach -n infra1_neutron_agents_container-de4c0565<\/span><\/pre>\n<p class=\"c0\">Once attached, we list the Linux namespaces using the following command:<\/p>\n<pre class=\"prettyprint\">root@infra1-neutron-agents-container-de4c0565:~# ip netns list\r\nqdhcp-d5b08c40-3b9e-4d6e-872b-7e8ce1553703 (id: 5)\r\nqrouter-215981d1-32bc-41fd-a13a-27dbe7ecb63f (id: 4)\r\nqdhcp-579d7378-84ad-45d5-882d-e052bab9595e (id: 3)\r\nqdhcp-33ad9c28-5884-41e1-97c0-05886740b5c1 (id: 1)\r\nqdhcp-423b2b1b-5591-4861-baab-64e9fef84f47 (id: 2)<\/pre>\n<p class=\"c0\">Based upon our previous bridge name <span class=\"codeinline\">brq423b2b1b-55<\/span> we can correlate this to the name space &#8211; \u00a0<span class=\"codeinline\">qdhcp-423b2b1b-5591-4861-baab-64e9fef84f47<\/span>.<\/p>\n<p class=\"c0\"><span class=\"c1\">With this namespace identifer we can run commands against the namespace, in order to help us with our troubleshooting. First, let us look at the IP address, this should be 192.168.1.2.<\/span><\/p>\n<pre class=\"prettyprint\">root@infra1-neutron-agents-container-de4c0565:~# ip netns exec qdhcp-423b2b1b-5591-4861-baab-64e9fef84f47 ifconfig\r\nlo        Link encap:Local Loopback\r\n          inet addr:127.0.0.1  Mask:255.0.0.0\r\n          inet6 addr: ::1\/128 Scope:Host\r\n          UP LOOPBACK RUNNING  MTU:65536  Metric:1\r\n          RX packets:0 errors:0 dropped:0 overruns:0 frame:0\r\n          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0\r\n          collisions:0 txqueuelen:1\r\n          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)\r\n\r\nns-e2e1c16d-6e Link encap:Ethernet  HWaddr fa:16:3e:82:97:6e\r\n          inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0\r\n          inet6 addr: fe80::f816:3eff:fe82:976e\/64 Scope:Link\r\n          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1\r\n          RX packets:160 errors:0 dropped:0 overruns:0 frame:0\r\n          TX packets:97 errors:0 dropped:0 overruns:0 carrier:0\r\n          collisions:0 txqueuelen:1000\r\n          RX bytes:12228 (12.2 KB)  TX bytes:7988 (7.9 KB)<\/pre>\n<p class=\"c0\"><span class=\"c1\">Well, the IP looks correct. Lets run another tcpdump, whilst running a curl on our guest.<\/span><\/p>\n<pre class=\"prettyprint\">tcpdump: verbose output suppressed, use -v or -vv for full protocol decode\r\nlistening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes\r\n22:14:13.529666 IP 192.168.1.50.49515 &gt; 169.254.169.254.80: Flags [S], seq 851007280, win 14100, options [mss 1410,sackOK,TS val 78147174 ecr 0,nop,wscale 2], length 0\r\n22:14:13.529699 IP 169.254.169.254.80 &gt; 192.168.1.50.49515: Flags [R.], seq 0, ack 851007281, win 0, length 0<\/pre>\n<p class=\"c0 c2\">Interesting. So let us take a moment. The SYN from the 3WHS is making it all the way to the necessary Linux Bridge on the compute node. The SYN is then making it to the DHCP agent and its corresponding namespace, but still there is something sendin a RST and preventing the TCP connection to form. We at least know the path is good, but what next? Let us check that the metadata service proxy is actually listening on the required port.<\/p>\n<pre class=\"prettyprint\">root@infra1-neutron-agents-container-de4c0565:~# ip netns exec qdhcp-423b2b1b-5591-4861-baab-64e9fef84f47 netstat -anp\r\nActive Internet connections (servers and established)\r\nProto Recv-Q Send-Q Local Address           Foreign Address         State       PID\/Program name\r\ntcp        0      0 192.168.1.2:53          0.0.0.0:*               LISTEN      1921\/dnsmasq\r\ntcp        0      0 169.254.169.254:53      0.0.0.0:*               LISTEN      1921\/dnsmasq\r\ntcp6       0      0 fe80::f816:3eff:fe82:53 :::*                    LISTEN      1921\/dnsmasq\r\nudp        0      0 192.168.1.2:47830       10.0.3.1:53             ESTABLISHED 1642\/tcpdump\r\nudp        0      0 192.168.1.2:53          0.0.0.0:*                           1921\/dnsmasq\r\nudp        0      0 169.254.169.254:53      0.0.0.0:*                           1921\/dnsmasq\r\nudp        0      0 0.0.0.0:67              0.0.0.0:*                           1921\/dnsmasq\r\nudp6       0      0 fe80::f816:3eff:fe82:53 :::*                                1921\/<\/pre>\n<p class=\"c0\"><span class=\"c1\">Wait, port 80 isnt listening. Let us restart the services, and recheck.<\/span><\/p>\n<pre class=\"prettyprint\">root@infra1-neutron-agents-container-de4c0565:~# service neutron-dhcp-agent restart\r\nroot@infra1-neutron-agents-container-de4c0565:~# service neutron-metadata-agent restart<\/pre>\n<p>root@infra1-neutron-agents-container-de4c0565:~# ip netns exec qdhcp-423b2b1b-5591-4861-baab-64e9fef84f47 netstat -anp Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID\/Program name tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1764\/python tcp 0 0 192.168.1.2:53 0.0.0.0:* LISTEN 1921\/dnsmasq tcp 0 0 169.254.169.254:53 0.0.0.0:* LISTEN 1921\/dnsmasq tcp6 0 0 fe80::f816:3eff:fe82:53 :::* LISTEN 1921\/dnsmasq udp 0 0 192.168.1.2:47830 10.0.3.1:53 ESTABLISHED 1642\/tcpdump udp 0 0 192.168.1.2:53 0.0.0.0:* 1921\/dnsmasq udp 0 0 169.254.169.254:53 0.0.0.0:* 1921\/dnsmasq udp 0 0 0.0.0.0:67 0.0.0.0:* 1921\/dnsmasq udp6 0 0 fe80::f816:3eff:fe82:53 :::* 1921\/dnsmasq<\/p>\n<p class=\"c0\">Now port 80 is back up, we can now see if the metaservice is accessable from our guest.<\/p>\n<h2 class=\"c0\"><span class=\"ez-toc-section\" id=\"Validation\"><\/span><span class=\"c1\">Validation<\/span><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<pre class=\"prettyprint\">$ curl -v \r\n&gt; GET \/ HTTP\/1.1\r\n&gt; User-Agent: curl\/7.24.0 (x86_64-pc-linux-gnu) libcurl\/7.24.0 OpenSSL\/1.0.0j zlib\/1.2.6\r\n&gt; Host: 169.254.169.254\r\n&gt; Accept: *\/*\r\n&gt;\r\n&lt; HTTP\/1.1 200 OK\r\n&lt; Content-Type: text\/plain; charset=UTF-8\r\n&lt; Content-Length: 98\r\n&lt; Date: Wed, 06 Sep 2017 20:32:51 GMT\r\n&lt;\r\n1.0\r\n2007-01-19\r\n2007-03-01\r\n2007-08-29\r\n2007-10-10\r\n2007-12-15\r\n2008-02-01\r\n2008-09-01\r\n2009-04-04\r\nlatest$<\/pre>\n<p class=\"c0\"><span class=\"c1\">Great, we are now getting the result as expected.\u00a0<\/span><\/p>\n<h2 class=\"c0\"><span class=\"ez-toc-section\" id=\"Summary\"><\/span>Summary<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p class=\"c0\">Now, to be fair the fact that we saw a RST being sent back on the compute node did indicate that a port may not have been listening. However, as im sure you will appreciate, the ability to troubleshoot and debug through the various points on an OpenStack system is extremely valuable.<\/p>\n<h2 class=\"c0\"><span class=\"ez-toc-section\" id=\"References\"><\/span>References<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p class=\"c5\"><a id=\"ftnt1\" href=\"#ftnt_ref1\">[1]<\/a><span class=\"c10\">\u00a0&#8220;Networking configuration options &#8211; OpenStack Documentation.&#8221; 12 Jun. 2017, <\/span><span class=\"c9\"><a class=\"c11\" href=\"https:\/\/www.google.com\/url?q=https:\/\/docs.openstack.org\/mitaka\/config-reference\/networking\/networking_options_reference.html&amp;sa=D&amp;ust=1504768276421000&amp;usg=AFQjCNEb7dqdNhbHMmIQKXQTpdjM4Ycb8A\">https:\/\/docs.openstack.org\/mitaka\/config-reference\/networking\/networking_options_reference.html<\/a><\/span><span class=\"c8\">. Accessed 6 Sep. 2017.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&nbsp; Introduction After troubleshooting a recent issue with accessing meta services from a guest instance, and jumping through the various steps within the path, I soon realised &#8211; this would make for a great article. Issue So first of all let&#8217;s look at the issue. The issue was pretty basic. Quite simply I was unable &#8230; <a title=\"Troubleshooting Connectivity to the Neutron Metadata Proxy\" class=\"read-more\" href=\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\" aria-label=\"Read more about Troubleshooting Connectivity to the Neutron Metadata Proxy\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":1136,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[74],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Troubleshooting Connectivity to the Neutron Metadata Proxy - Fir3net<\/title>\n<meta name=\"description\" content=\"&nbsp; Introduction After troubleshooting a recent issue with accessing meta services from a guest instance, and jumping through the various steps within\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Troubleshooting Connectivity to the Neutron Metadata Proxy - Fir3net\" \/>\n<meta property=\"og:description\" content=\"&nbsp; Introduction After troubleshooting a recent issue with accessing meta services from a guest instance, and jumping through the various steps within\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\" \/>\n<meta property=\"og:site_name\" content=\"Fir3net\" \/>\n<meta property=\"article:published_time\" content=\"2017-09-05T23:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-07-31T15:41:09+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.fir3net.com\/wp-content\/uploads\/2017\/09\/images_meta-proxy-icon.png\" \/>\n\t<meta property=\"og:image:width\" content=\"374\" \/>\n\t<meta property=\"og:image:height\" content=\"374\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Rick Donato\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Rick Donato\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\"},\"author\":{\"name\":\"Rick Donato\",\"@id\":\"https:\/\/www.fir3net.com\/#\/schema\/person\/ab35009601b7687ee1c5310be6038037\"},\"headline\":\"Troubleshooting Connectivity to the Neutron Metadata Proxy\",\"datePublished\":\"2017-09-05T23:00:00+00:00\",\"dateModified\":\"2021-07-31T15:41:09+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\"},\"wordCount\":793,\"publisher\":{\"@id\":\"https:\/\/www.fir3net.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fir3net.com\/wp-content\/uploads\/2017\/09\/images_meta-proxy-icon.png\",\"articleSection\":[\"OpenStack\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\",\"url\":\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\",\"name\":\"Troubleshooting Connectivity to the Neutron Metadata Proxy - Fir3net\",\"isPartOf\":{\"@id\":\"https:\/\/www.fir3net.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fir3net.com\/wp-content\/uploads\/2017\/09\/images_meta-proxy-icon.png\",\"datePublished\":\"2017-09-05T23:00:00+00:00\",\"dateModified\":\"2021-07-31T15:41:09+00:00\",\"description\":\"&nbsp; Introduction After troubleshooting a recent issue with accessing meta services from a guest instance, and jumping through the various steps within\",\"breadcrumb\":{\"@id\":\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html#primaryimage\",\"url\":\"https:\/\/www.fir3net.com\/wp-content\/uploads\/2017\/09\/images_meta-proxy-icon.png\",\"contentUrl\":\"https:\/\/www.fir3net.com\/wp-content\/uploads\/2017\/09\/images_meta-proxy-icon.png\",\"width\":374,\"height\":374},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.fir3net.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Cloud\",\"item\":\"https:\/\/www.fir3net.com\/cloud\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"OpenStack\",\"item\":\"https:\/\/www.fir3net.com\/cloud\/openstack\"},{\"@type\":\"ListItem\",\"position\":4,\"name\":\"Troubleshooting Connectivity to the Neutron Metadata Proxy\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.fir3net.com\/#website\",\"url\":\"https:\/\/www.fir3net.com\/\",\"name\":\"Fir3net\",\"description\":\"Keeping you in the know\",\"publisher\":{\"@id\":\"https:\/\/www.fir3net.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.fir3net.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.fir3net.com\/#organization\",\"name\":\"Fir3net\",\"url\":\"https:\/\/www.fir3net.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fir3net.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.fir3net.com\/wp-content\/uploads\/Fir3net-Background-Logo-compressed.png\",\"contentUrl\":\"https:\/\/www.fir3net.com\/wp-content\/uploads\/Fir3net-Background-Logo-compressed.png\",\"width\":390,\"height\":88,\"caption\":\"Fir3net\"},\"image\":{\"@id\":\"https:\/\/www.fir3net.com\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.fir3net.com\/#\/schema\/person\/ab35009601b7687ee1c5310be6038037\",\"name\":\"Rick Donato\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fir3net.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d75d69a54c0ca3b32c24c3a9703b623c?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d75d69a54c0ca3b32c24c3a9703b623c?s=96&d=mm&r=g\",\"caption\":\"Rick Donato\"},\"description\":\"Rick Donato is a Network Automation Architect\/Evangelist and the founder of Packet Coders.\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Troubleshooting Connectivity to the Neutron Metadata Proxy - Fir3net","description":"&nbsp; Introduction After troubleshooting a recent issue with accessing meta services from a guest instance, and jumping through the various steps within","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html","og_locale":"en_US","og_type":"article","og_title":"Troubleshooting Connectivity to the Neutron Metadata Proxy - Fir3net","og_description":"&nbsp; Introduction After troubleshooting a recent issue with accessing meta services from a guest instance, and jumping through the various steps within","og_url":"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html","og_site_name":"Fir3net","article_published_time":"2017-09-05T23:00:00+00:00","article_modified_time":"2021-07-31T15:41:09+00:00","og_image":[{"width":374,"height":374,"url":"https:\/\/www.fir3net.com\/wp-content\/uploads\/2017\/09\/images_meta-proxy-icon.png","type":"image\/jpeg"}],"author":"Rick Donato","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Rick Donato","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html#article","isPartOf":{"@id":"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html"},"author":{"name":"Rick Donato","@id":"https:\/\/www.fir3net.com\/#\/schema\/person\/ab35009601b7687ee1c5310be6038037"},"headline":"Troubleshooting Connectivity to the Neutron Metadata Proxy","datePublished":"2017-09-05T23:00:00+00:00","dateModified":"2021-07-31T15:41:09+00:00","mainEntityOfPage":{"@id":"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html"},"wordCount":793,"publisher":{"@id":"https:\/\/www.fir3net.com\/#organization"},"image":{"@id":"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html#primaryimage"},"thumbnailUrl":"https:\/\/www.fir3net.com\/wp-content\/uploads\/2017\/09\/images_meta-proxy-icon.png","articleSection":["OpenStack"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html","url":"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html","name":"Troubleshooting Connectivity to the Neutron Metadata Proxy - Fir3net","isPartOf":{"@id":"https:\/\/www.fir3net.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html#primaryimage"},"image":{"@id":"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html#primaryimage"},"thumbnailUrl":"https:\/\/www.fir3net.com\/wp-content\/uploads\/2017\/09\/images_meta-proxy-icon.png","datePublished":"2017-09-05T23:00:00+00:00","dateModified":"2021-07-31T15:41:09+00:00","description":"&nbsp; Introduction After troubleshooting a recent issue with accessing meta services from a guest instance, and jumping through the various steps within","breadcrumb":{"@id":"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html#primaryimage","url":"https:\/\/www.fir3net.com\/wp-content\/uploads\/2017\/09\/images_meta-proxy-icon.png","contentUrl":"https:\/\/www.fir3net.com\/wp-content\/uploads\/2017\/09\/images_meta-proxy-icon.png","width":374,"height":374},{"@type":"BreadcrumbList","@id":"https:\/\/www.fir3net.com\/Cloud\/Openstack\/troubleshooting-connectivity-to-the-neutron-metadata-proxy.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.fir3net.com\/"},{"@type":"ListItem","position":2,"name":"Cloud","item":"https:\/\/www.fir3net.com\/cloud"},{"@type":"ListItem","position":3,"name":"OpenStack","item":"https:\/\/www.fir3net.com\/cloud\/openstack"},{"@type":"ListItem","position":4,"name":"Troubleshooting Connectivity to the Neutron Metadata Proxy"}]},{"@type":"WebSite","@id":"https:\/\/www.fir3net.com\/#website","url":"https:\/\/www.fir3net.com\/","name":"Fir3net","description":"Keeping you in the know","publisher":{"@id":"https:\/\/www.fir3net.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.fir3net.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.fir3net.com\/#organization","name":"Fir3net","url":"https:\/\/www.fir3net.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fir3net.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.fir3net.com\/wp-content\/uploads\/Fir3net-Background-Logo-compressed.png","contentUrl":"https:\/\/www.fir3net.com\/wp-content\/uploads\/Fir3net-Background-Logo-compressed.png","width":390,"height":88,"caption":"Fir3net"},"image":{"@id":"https:\/\/www.fir3net.com\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.fir3net.com\/#\/schema\/person\/ab35009601b7687ee1c5310be6038037","name":"Rick Donato","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fir3net.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d75d69a54c0ca3b32c24c3a9703b623c?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d75d69a54c0ca3b32c24c3a9703b623c?s=96&d=mm&r=g","caption":"Rick Donato"},"description":"Rick Donato is a Network Automation Architect\/Evangelist and the founder of Packet Coders."}]}},"_links":{"self":[{"href":"https:\/\/www.fir3net.com\/wp-json\/wp\/v2\/posts\/1137"}],"collection":[{"href":"https:\/\/www.fir3net.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.fir3net.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.fir3net.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fir3net.com\/wp-json\/wp\/v2\/comments?post=1137"}],"version-history":[{"count":0,"href":"https:\/\/www.fir3net.com\/wp-json\/wp\/v2\/posts\/1137\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.fir3net.com\/wp-json\/wp\/v2\/media\/1136"}],"wp:attachment":[{"href":"https:\/\/www.fir3net.com\/wp-json\/wp\/v2\/media?parent=1137"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.fir3net.com\/wp-json\/wp\/v2\/categories?post=1137"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.fir3net.com\/wp-json\/wp\/v2\/tags?post=1137"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}