SLES 12 network issue

SUSE Linux Enterprise Server SLES for System Z
1

I am having a problem with a Linux guest that I upgraded in place from SLES 11 SP4 to SLES 12 SP3. The problem is that I cannot PuTTY into the server from my network (using a NAT address) nor from my clients’ network using the real address. The PuTTY sessions time out. I believe that the request is getting to the SLES 12 system and is being rejected. SUSE Support (I have opened to SR’s for this) says that the OS is working fine as I can SSH into this server from a server running on the same z/VM system on the same 10.17.1.156/26 network.

In order to minimize the application outage time for the PROD server, I decided to upgrade a clone of PROD’s boot volume using a new, temporary, server. The upgrade was performed by creating a new virtual server, TEST, and restoring the PROD boot volume from tape (from a clean backup taken while PROD was shut down) to a new UCB address. I used the SLES12 REXX exec and specified a profile called TEST where the only difference from PROD is the IP address. The upgrade went smoothly and took about two hours to complete, with most of the time needed to modify about 25 config files where the upgrade created .rpmnew files.

When the starter linux system completed and rebooted the server, I was unable to PuTTY into the server using the new IP address. After lots of trial and error I opened an SR with support. They said that since ssh worked from another server, that there was nothing wrong with the SLES 12 system itself, that my network is the problem. After more testing I still thought the problem was with SLES 12 and opened a new SR and referred to the previous one. Support still says that SLES 12 isn’t the problem.

As a test, I shut down a SLES 12 server that I am able PuTTY into (this server was upgraded to SLES 12 in mid-September and is at a different package level than TEST). I then modified /etc/sysconfig/netowrk/ifgcfg-eth0 on TEST and made it look exactly like the file on the server I just shut down. I shut down TEST and IPL’ed the guest (as opposed to rebooting the server). I verified from the console that TEST was using the IP address from the SLES 12 server that I was able to access via PuTTY. I cannot access TEST with PuTTY (using the real IP address from the clients’ network, nor from my network using the NAT address).

I ran a tracert command from a Windows jump server on the client network. After the first hop was displayed by tracert I started seeing the following messages on the console for TEST:

2017-12-04T13:18:49.874194-06:00 wcs-mf-winxs-db2p kernel: IPv4: martian source 10.17.1.156 from 172.24.8.30, on dev eth0 2017-12-04T13:18:49.874220-06:00 wcs-mf-winxs-db2p kernel: ll header: 00000000: 02 00 00 00 02 d6 00 26 51 cc f9 41 08 00 ... ....&Q..A.. 2017-12-04T13:18:53.795374-06:00 wcs-mf-winxs-db2p kernel: IPv4: martian source 10.17.1.156 from 172.24.8.30, on dev eth0 2017-12-04T13:18:53.795398-06:00 wcs-mf-winxs-db2p kernel: ll header: 00000000: 02 00 00 00 02 d6 00 26 51 cc f9 41 08 00 ... ....&Q..A.. 2017-12-04T13:18:57.804186-06:00 wcs-mf-winxs-db2p kernel: IPv4: martian source 10.17.1.156 from 172.24.8.30, on dev eth0 2017-12-04T13:18:57.804212-06:00 wcs-mf-winxs-db2p kernel: ll header: 00000000: 02 00 00 00 02 d6 00 26 51 cc f9 41 08 00 ... ....&Q..A.. 2017-12-04T13:19:01.794178-06:00 wcs-mf-winxs-db2p kernel: IPv4: martian source 10.17.1.156 from 172.24.8.30, on dev eth0 2017-12-04T13:19:01.794194-06:00 wcs-mf-winxs-db2p kernel: ll header: 00000000: 02 00 00 00 02 d6 00 26 51 cc f9 41 08 00 ... ....&Q..A.. 2017-12-04T13:19:05.794200-06:00 wcs-mf-winxs-db2p kernel: IPv4: martian source 10.17.1.156 from 172.24.8.30, on dev eth0 2017-12-04T13:19:05.794226-06:00 wcs-mf-winxs-db2p kernel: ll header: 00000000: 02 00 00 00 02 d6 00 26 51 cc f9 41 08 00 ... ....&Q..A.. 2017-12-04T13:19:09.804197-06:00 wcs-mf-winxs-db2p kernel: IPv4: martian source 10.17.1.156 from 172.24.8.30, on dev eth0 2017-12-04T13:19:09.804220-06:00 wcs-mf-winxs-db2p kernel: ll header: 00000000: 02 00 00 00 02 d6 00 26 51 cc f9 41 08 00 ... ....&Q..A.. 2017-12-04T13:19:14.114174-06:00 wcs-mf-winxs-db2p kernel: IPv4: martian source 10.17.1.156 from 172.24.8.30, on dev eth0 2017-12-04T13:19:14.114201-06:00 wcs-mf-winxs-db2p kernel: ll header: 00000000: 02 00 00 00 02 d6 00 26 51 cc f9 41 08 00 ... ....&Q..A..

The jump server’s address is 172.24.8.30 and TEST’s IP is 10.17.1.156. This is what leads me to believe that the problem is with the SLES 12 system installed on TEST. Linux is displaying messages showing that the request is reaching the IP stack. Note that this problem exists whether or not SuSEFirewall2 is running. There isn’t anything about this request being logged in /var/log/firewall.

Any ideas on what I need to change to resolve this?

Harley

2

Disclaimer: I do not have a Z system, so I’m going to ignore everything
about it other than reboot/IPL which, to my limited mind, mean similar
things enough that I’ll just probably say “reboot” and hope you can help
me cover the gap there. :slight_smile:

First, nice reporting of the issue. I appreciate the time it took you to
write all of that, and think through it, and fight through SRs, etc.

On to the problem, I think I agree with each of you a bit; clearly some
packets are reaching your system, so the network outside of the system is
working correct to get the packets there. To get more information
regarding what is happening on the wire from this machine’s perspective I
would recommend using tcpdump:

sudo zypper install tcpdump  #in case it is not installed already
sudo /usr/sbin/tcpdump -n -s 0 -i any port 22

While that is running, go ahead and try to SSH in from some other system
that does not work. If you can do it from a system that does work, that
may be useful too. What we should see are packets going back and forth,
rather than just going one-way. tcpdump is neat because it works before
the Linux firewall (Netfilter) interferes, so even if it is blocked by the
host’s firewall (you said it is not) the packets should show up.

Another test we may be able to do is get more output from SSH during the
connection. If you have a Linux box that CANNOT SSH in, then use that
one; just add ‘-vvv’ (three ‘v’ characters) to get a ton of debugging
information from the client (passwords and keys are never shown as I recall):

ssh -vvv username@remote.box.goes.here

Some other useful information for us may be ip and routing information,
which is simple to get:

ip a  #ip address info
ip r  #routing information
ip neigh  #arp/neighbor information

In this case I m mostly interested in the address and routing information,
as I wonder if there may be a route missing to allow packets to get back
to the source properly. Do you have other systems, SLES 11 maybe, that
can be SSH’d-info, from which we can get the same commands’ output? It
could be useful for comparison purposes if nothing else.

–
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.

3

Ab,

For the most part, IPL = boot. Us mainframers still think the old way :D.

Here is the tcpdump output from the PuTTY session (fails):

reading from file from-putty.pcap, link-type LINUX_SLL (Linux cooked) 10:56:57.250880 IP 10.17.1.156.ssh > 10.17.1.163.36438: Flags [P.], seq 3141535555:3141535691, ack 2392637200, win 301, options [nop,nop,TS val 893702 ecr 156803946], length 136 10:56:57.250909 IP 10.17.1.163.36438 > 10.17.1.156.ssh: Flags [.], ack 136, win 661, options [nop,nop,TS val 156803947 ecr 893702], length 0 10:57:04.769580 IP 172.24.8.30.52930 > 10.17.1.156.ssh: Flags [S], seq 3987265288, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0 10:57:07.773934 IP 172.24.8.30.52930 > 10.17.1.156.ssh: Flags [S], seq 3987265288, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0 10:57:13.779932 IP 172.24.8.30.52930 > 10.17.1.156.ssh: Flags [S], seq 3987265288, win 8192, options [mss 1460,nop,nop,sackOK], length 0

IP 10.17.1.156 is the server that is having the issue.
IP 10.17.1.163 is the server where I issued ‘ssh -X root@10.17.1.156’ to get to Linux.
IP 172.24.8.30 is the Windows server where I attempted to connect via PuTTY.

Here is the tcpdump output form an SSH connection from another server running on the same VM as the server having the issue"

reading from file from-ssh.pcap, link-type LINUX_SLL (Linux cooked) 11:12:58.514541 IP 10.17.1.156.ssh > 10.17.1.163.36458: Flags [P.], seq 2483141602:2483141738, ack 1440035107, win 317, options [nop,nop,TS val 989828 ecr 156900073], length 136 11:12:58.514577 IP 10.17.1.163.36458 > 10.17.1.156.ssh: Flags [.], ack 136, win 471, options [nop,nop,TS val 156900073 ecr 989828], length 0 11:13:01.403400 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [S], seq 1421146552, win 14520, options [mss 1452,sackOK,TS val 975507037 ecr 0,nop,wscale 3], length 0 11:13:01.403484 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [S.], seq 3247949623, ack 1421146553, win 28960, options [mss 1460,sackOK,TS val 990117 ecr 975507037,nop,wscale 7], length 0 11:13:01.403805 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 1, win 1815, options [nop,nop,TS val 975507038 ecr 990117], length 0 11:13:01.404014 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 1:24, ack 1, win 1815, options [nop,nop,TS val 975507038 ecr 990117], length 23 11:13:01.404022 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [.], ack 24, win 227, options [nop,nop,TS val 990117 ecr 975507038], length 0 11:13:01.423467 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 1:22, ack 24, win 227, options [nop,nop,TS val 990119 ecr 975507038], length 21 11:13:01.423707 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 22, win 1815, options [nop,nop,TS val 975507040 ecr 990119], length 0 11:13:01.423869 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], seq 24:1464, ack 22, win 1815, options [nop,nop,TS val 975507040 ecr 990119], length 1440 11:13:01.423876 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 1464:1896, ack 22, win 1815, options [nop,nop,TS val 975507040 ecr 990119], length 432 11:13:01.423878 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [.], ack 1896, win 272, options [nop,nop,TS val 990119 ecr 975507040], length 0 11:13:01.424021 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 22:1006, ack 1896, win 272, options [nop,nop,TS val 990119 ecr 975507040], length 984 11:13:01.432116 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 1896:1944, ack 1006, win 2175, options [nop,nop,TS val 975507040 ecr 990119], length 48 11:13:01.443100 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 1006:1286, ack 1944, win 272, options [nop,nop,TS val 990121 ecr 975507040], length 280 11:13:01.486581 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 1286, win 2421, options [nop,nop,TS val 975507046 ecr 990121], length 0 11:13:03.071856 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 1944:1960, ack 1286, win 2421, options [nop,nop,TS val 975507204 ecr 990121], length 16 11:13:03.111578 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [.], ack 1960, win 272, options [nop,nop,TS val 990288 ecr 975507204], length 0 11:13:03.111664 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 1960:2016, ack 1286, win 2421, options [nop,nop,TS val 975507208 ecr 990288], length 56 11:13:03.111672 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [.], ack 2016, win 272, options [nop,nop,TS val 990288 ecr 975507208], length 0 11:13:03.111731 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 1286:1342, ack 2016, win 272, options [nop,nop,TS val 990288 ecr 975507208], length 56 11:13:03.111879 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 1342, win 2421, options [nop,nop,TS val 975507208 ecr 990288], length 0 11:13:03.111980 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 2016:2088, ack 1342, win 2421, options [nop,nop,TS val 975507208 ecr 990288], length 72 11:13:03.112225 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 1342:1414, ack 2088, win 272, options [nop,nop,TS val 990288 ecr 975507208], length 72 11:13:03.114994 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 2088:2176, ack 1414, win 2421, options [nop,nop,TS val 975507209 ecr 990288], length 88 11:13:03.115578 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 1414:1486, ack 2176, win 272, options [nop,nop,TS val 990288 ecr 975507209], length 72 11:13:03.152534 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 1486, win 2421, options [nop,nop,TS val 975507213 ecr 990288], length 0 11:13:05.543461 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 2176:2264, ack 1486, win 2421, options [nop,nop,TS val 975507452 ecr 990288], length 88 11:13:05.545216 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 1486:1542, ack 2264, win 272, options [nop,nop,TS val 990531 ecr 975507452], length 56 11:13:05.545250 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 1542, win 2421, options [nop,nop,TS val 975507452 ecr 990531], length 0 11:13:05.545268 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 2264:2352, ack 1542, win 2421, options [nop,nop,TS val 975507452 ecr 990531], length 88 11:13:05.545415 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 1542:1582, ack 2352, win 272, options [nop,nop,TS val 990531 ecr 975507452], length 40 11:13:05.545547 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 2352:2480, ack 1582, win 2421, options [nop,nop,TS val 975507452 ecr 990531], length 128 11:13:05.549783 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 1582:2534, ack 2480, win 294, options [nop,nop,TS val 990531 ecr 975507452], length 952 11:13:05.582522 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 2534, win 2781, options [nop,nop,TS val 975507456 ecr 990531], length 0 11:13:05.582544 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 2534:2590, ack 2480, win 294, options [nop,nop,TS val 990535 ecr 975507456], length 56 11:13:05.582695 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 2590, win 2781, options [nop,nop,TS val 975507456 ecr 990535], length 0 11:13:05.582742 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 2480:3024, ack 2590, win 2781, options [nop,nop,TS val 975507456 ecr 990535], length 544 11:13:05.583165 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 2590:2710, ack 3024, win 317, options [nop,nop,TS val 990535 ecr 975507456], length 120 11:13:05.584766 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 2710:2814, ack 3024, win 317, options [nop,nop,TS val 990535 ecr 975507456], length 104 11:13:05.584790 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 2814, win 2781, options [nop,nop,TS val 975507456 ecr 990535], length 0 11:13:05.639840 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 2814:2902, ack 3024, win 317, options [nop,nop,TS val 990540 ecr 975507456], length 88 11:13:05.672520 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 2902, win 2781, options [nop,nop,TS val 975507465 ecr 990540], length 0 11:13:06.239160 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 3024:3064, ack 2902, win 2781, options [nop,nop,TS val 975507521 ecr 990540], length 40 11:13:06.239453 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 2902:2942, ack 3064, win 317, options [nop,nop,TS val 990600 ecr 975507521], length 40 11:13:06.239565 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 2942, win 2781, options [nop,nop,TS val 975507521 ecr 990600], length 0 11:13:06.374758 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 3064:3104, ack 2942, win 2781, options [nop,nop,TS val 975507535 ecr 990600], length 40 11:13:06.374877 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 2942:2982, ack 3104, win 317, options [nop,nop,TS val 990614 ecr 975507535], length 40 11:13:06.374996 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 2982, win 2781, options [nop,nop,TS val 975507535 ecr 990614], length 0 11:13:06.505230 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 3104:3144, ack 2982, win 2781, options [nop,nop,TS val 975507548 ecr 990614], length 40 11:13:06.505344 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 2982:3022, ack 3144, win 317, options [nop,nop,TS val 990627 ecr 975507548], length 40 11:13:06.505401 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 3022, win 2781, options [nop,nop,TS val 975507548 ecr 990627], length 0 11:13:06.671049 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 3144:3184, ack 3022, win 2781, options [nop,nop,TS val 975507564 ecr 990627], length 40 11:13:06.671151 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 3022:3062, ack 3184, win 317, options [nop,nop,TS val 990644 ecr 975507564], length 40 11:13:06.671205 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 3062, win 2781, options [nop,nop,TS val 975507564 ecr 990644], length 0 11:13:06.799167 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 3184:3224, ack 3062, win 2781, options [nop,nop,TS val 975507577 ecr 990644], length 40 11:13:06.800044 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 3062:3230, ack 3224, win 317, options [nop,nop,TS val 990656 ecr 975507577], length 168 11:13:06.800063 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 3230:3310, ack 3224, win 317, options [nop,nop,TS val 990656 ecr 975507577], length 80 11:13:06.803578 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 3230, win 3027, options [nop,nop,TS val 975507578 ecr 990656], length 0 11:13:06.803590 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 3310, win 3027, options [nop,nop,TS val 975507578 ecr 990656], length 0 11:13:06.803616 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 3224:3264, ack 3310, win 3027, options [nop,nop,TS val 975507578 ecr 990656], length 40 11:13:06.803629 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 3264:3336, ack 3310, win 3027, options [nop,nop,TS val 975507578 ecr 990656], length 72 11:13:06.803643 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [.], ack 3336, win 317, options [nop,nop,TS val 990657 ecr 975507578], length 0 11:13:06.803653 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [F.], seq 3336, ack 3310, win 3027, options [nop,nop,TS val 975507578 ecr 990656], length 0 11:13:06.805795 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [F.], seq 3310, ack 3337, win 317, options [nop,nop,TS val 990657 ecr 975507578], length 0 11:13:06.815075 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 3311, win 3027, options [nop,nop,TS val 975507579 ecr 990657], length 0

IP 10.17.1.156 is the server that is having the issue.
IP 10.17.1.139 is the server where I issued ‘ssh -X root@10.17.1.156’ to get to Linux.

Here is the output from ‘ssh -vvv -X root@10.17.1.156’.

[CODE]OpenSSH_6.6.1, OpenSSL 0.9.8j-fips 07 Jan 2009
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 20: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 10.17.1.156 [10.17.1.156] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2
debug1: match: OpenSSH_7.2 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host “10.17.1.156” from file “/root/.ssh/known_hosts”
debug3: load_hostkeys: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: kex_parse_kexinit: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup hmac-sha1-etm@openssh.com
debug1: kex: server->client aes128-ctr hmac-sha1-etm@openssh.com none
debug2: mac_setup: setup hmac-sha1-etm@openssh.com
debug1: kex: client->server aes128-ctr hmac-sha1-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 97:05:65:80:55:a8:c5:23:de:c5:57:16:f0:ea:28:ed [MD5]
debug3: load_hostkeys: loading entries for host “10.17.1.156” from file “/root/.ssh/known_hosts”
debug3: load_hostkeys: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug1: Host ‘10.17.1.156’ is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /root/.ssh/id_rsa ((nil)),
debug2: key: /root/.ssh/id_dsa ((nil)),
debug2: key: /root/.ssh/id_ecdsa ((nil)),
debug2: key: /root/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,keyboard-interactive
debug3: start over, passed a different list publickey,keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug3: no such identity: /root/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ecdsa
debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ed25519
debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password:
debug3: packet_send2: adding 32 (len 18 padlen 14 extra_pad 64)
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 0
debug3: packet_send2: adding 48 (len 6 padlen 10 extra_pad 64)
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to 10.17.1.156 ([10.17.1.156]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug3: Ignored env LESSKEY
debug3: Ignored env NNTPSERVER
debug3: Ignored env INFODIR
debug3: Ignored env MANPATH
debug3: Ignored env HOSTNAME
debug3: Ignored env XKEYSYMDB
debug3: Ignored env HOST
debug3: Ignored env SHELL
debug3: Ignored env TERM
debug3: Ignored env PROFILEREAD
debug3: Ignored env HISTSIZE
debug3: Ignored env MORE
debug3: Ignored env USER
debug3: Ignored env LS_COLORS
debug3: Ignored env XNLSPATH
debug3: Ignored env ENV
debug3: Ignored env HOSTTYPE
debug3: Ignored env FROM_HEADER
debug3: Ignored env PAGER
debug3: Ignored env CSHEDIT
debug3: Ignored env XDG_CONFIG_DIRS
debug3: Ignored env MINICOM
debug3: Ignored env MAIL
debug3: Ignored env PATH
debug3: Ignored env CPU
debug3: Ignored env INPUTRC
debug3: Ignored env PWD
debug1: Sending env LANG = POSIX
debug2: channel 0: request env confirm 0
debug3: Ignored env PYTHONSTARTUP
debug3: Ignored env QT_SYSTEM_DIR
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env LESS_ADVANCED_PREPROCESSOR
debug3: Ignored env OSTYPE
debug3: Ignored env LS_OPTIONS
debug3: Ignored env XCURSOR_THEME
debug3: Ignored env WINDOWMANAGER
debug3: Ignored env LESS
debug3: Ignored env MACHTYPE
debug3: Ignored env LOGNAME
debug3: Ignored env XDG_DATA_DIRS
debug1: Sending env LC_CTYPE = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env LESSOPEN
debug3: Ignored env INFOPATH
debug3: Ignored env LESSCLOSE
debug3: Ignored env G_BROKEN_FILENAMES
debug3: Ignored env COLORTERM
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Tue Dec 5 11:16:48 2017 from 10.17.1.139
wcs-mf-winxs-db2p:~ # exitdebug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug2: channel 0: rcvd eow
debug2: channel 0: close_read
debug2: channel 0: input open → closed
debug2: channel 0: rcvd eof
debug2: channel 0: output open → drain
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close

logout
debug3: channel 0: will not send data after close
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain → closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

Connection to 10.17.1.156 closed.
Transferred: sent 2952, received 2868 bytes, in 3.4 seconds
Bytes per second: sent 870.7, received 845.9
debug1: Exit status 0
[/CODE]

From the server having the issue:

wcs-mf-winxs-db2p:/tmp # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 02:00:00:00:02:d9 brd ff:ff:ff:ff:ff:ff inet 10.17.1.156/26 brd 10.17.1.191 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::ff:fe00:2d9/64 scope link valid_lft forever preferred_lft forever wcs-mf-winxs-db2p:/tmp # ip r 10.17.1.128/26 dev eth0 proto kernel scope link src 10.17.1.156 wcs-mf-winxs-db2p:/tmp # ip neigh 10.17.1.139 dev eth0 lladdr 02:00:00:00:00:24 STALE 10.17.1.189 dev eth0 lladdr 00:22:55:7a:12:c1 STALE 10.17.1.190 dev eth0 lladdr 00:26:51:cc:f9:41 STALE 10.17.1.163 dev eth0 lladdr 02:00:00:00:02:74 DELAY

From a server on the same hardware and same network that does not have the issue"

wcs-mf-smt:~ # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 brd 127.255.255.255 scope host lo inet 127.0.0.2/8 brd 127.255.255.255 scope host secondary lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1492 qdisc pfifo_fast state UNKNOWN qlen 1000 link/ether 02:00:00:00:00:24 brd ff:ff:ff:ff:ff:ff inet 10.17.1.139/26 brd 10.17.1.191 scope global eth0 inet6 fe80::ff:fe00:24/64 scope link valid_lft forever preferred_lft forever wcs-mf-smt:~ # ip r default via 10.17.1.129 dev eth0 10.17.1.128/26 dev eth0 proto kernel scope link src 10.17.1.139 127.0.0.0/8 dev lo scope link 169.254.0.0/16 dev eth0 scope link wcs-mf-smt:~ # ip neigh 10.17.1.129 dev eth0 lladdr 00:00:0c:9f:f2:00 REACHABLE

Harley

4

Unless I am missing something, your broken box does not have a default
route. Your ‘ip r’ information shows one route, for the local network,
but that does not help at all when you want to get packets from here to
any other box off the local network, including back to where packets
arriving from there originated.

Compare the output from your broken and working boxes, respectively:

Broken:

wcs-mf-winxs-db2p:/tmp # ip r
10.17.1.128/26 dev eth0  proto kernel  scope link  src 10.17.1.156

Working:

wcs-mf-smt:~ # ip r
default via 10.17.1.129 dev eth0
10.17.1.128/26 dev eth0  proto kernel  scope link  src 10.17.1.139
127.0.0.0/8 dev lo  scope link
169.254.0.0/16 dev eth0  scope link

Try adding the default route of 100.17.1.129 to your broken box. On non-Z
systems I do this with ‘yast lan’, like this:

sudo /sbin/yast lan

In there you should see a tab for routing, and in there is a spot for an
IPv4 default gateway. Put in the IP address, save/Finish, and then test
again.

–
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.

5

I think that you mis-typed as there isn’t a 100.17.1.129 it is 10.17.1.129.

/etc/sysconfig/network/routes contains

default 10.17.1.129 - -

On the Routing tab of ‘yast lan’:
Default IPv4 Gateway is set to 10.17.1.129.
Default IPv6 Gateway is blank.
The Routing Table box is empty.
Enable IPv4 Forwarding is unchecked.
Enable IPv6 Forwarding is unchecked.

Not sure where to make the change you suggested.

6

Ab,

I was also working this issue with the Network team here (the router and firewall folks). One of them found that the default gateway was not being picked up from /etc/sysconfig/network/routes. He had me run ‘netstat -rnv’ and it displayed:

wcs-mf-winxs-db2p:/tmp # netstat -rnv Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.17.1.128 0.0.0.0 255.255.255.192 U 0 0 0 eth0

He then had me temporarily add a gateway by issuing ‘route add default gw 10.17.1.129’. I was then able to PuTTY into the server.

Any idea how to make this permanent (i.e. without coding a script to issue the command at boot time)? I will continue to research how to accomplish that.

Harley

7

Yes, exactly; I’m glad the same conclusion was reached.

It should already be permanent, per the configuration files, but it seems
the networking configuration or stack is a bit off. I have seen routes
NOT-add properly when the required NIC was not yet fully up, but I presume
you are using static IP addresses, so that should be unlikely. I’ve also
seen odd things happen when there was a duplicate IP on the network
somewhere, as the system may try to avoid conflicts for you since those
are generally bad; any chance of that situation happening lately?

Do you see anything useful when trying to get the status of the network
service?

sudo systemctl status network

For fun, while on the box physically-ish, could you restart your
networking to see if it comes up properly now that the rest of the system
is already working/booted?

sudo systemctl restart network

As a note, while it is probably generally fine to do, I’ve seldom worked
with the files under /etc/sysconfig/network/ directly, and would generally
recommend using Yast to configure things when doing so manually. Yast is
pretty robust, particularly around networking, and uses those files
itself; since I have had great luck using Yast, and seldom any problems
like this, it makes me wonder if some of the magical bits may be unhappy
with your changes for some pesky reason. To be fair, you already know I
am not a Z-series expert, so maybe issues there happen more-often in these
cases and all of my experience is for naught.

The route command basically shows the same thing as the ‘ip’ command, so
that’s good, and adding a route manually via ip (or route) does what the
system should be doing behind the scenes. Hopefully we can find something
in the logs (above) that will help us figure things out.

–
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.

8

Ab,

Your asking for info from ‘sudo systemctl status network’ showed me what my problem is.

In order to clone the disk for the server I needed to have it come up with a new IP address as the ‘real’ server is up an running. I saved a copy of ifcfg-eth0 in /etc/sysconfig/network and called it ifcfg-eth0.save.

[CODE]wcs-mf-winxs-db2p:~ # systemctl status network
â wicked.service - wicked managed network interfaces
Loaded: loaded (/usr/lib/systemd/system/wicked.service; enabled; vendor preset: disabled)
Active: active (exited) since Wed 2017-12-06 07:53:35 CST; 14s ago
Process: 1884 ExecStart=/usr/sbin/wicked --systemd ifup all (code=exited, status=0/SUCCESS)
Main PID: 1884 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 512)
CGroup: /system.slice/wicked.service

Dec 06 07:53:04 wcs-mf-winxs-db2p systemd[1]: Starting wicked managed network interfaces…
Dec 06 07:53:35 wcs-mf-winxs-db2p wicked[1884]: lo up
Dec 06 07:53:35 wcs-mf-winxs-db2p wicked[1884]: eth0 up
Dec 06 07:53:35 wcs-mf-winxs-db2p wicked[1884]: eth0.save no-device
Dec 06 07:53:35 wcs-mf-winxs-db2p systemd[1]: Started wicked managed network interfaces.
[/CODE]

Seeing the ‘eth0.save’ in the list rang a bell where (about 10 years ago) we had weird network issues upgrading to SLES 10 when we named a saved copy as filename.save. I renamed ifcfg-eth0.save to save.ifcfg-eth0, rebooted, and tested. I was able to PuTTY in without issuing the temporary command to define the gateway.

Thank you very much for your assistance with this.

Harley

9

Well that’s just great! Thank-you for posting back how you found the
issue as that is one worth remembering, and since memories are imperfect,
this is worth indexing via Google.

–
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.