Get salt to use /etc/sysconfig/proxy

[Rick]

I’ve been trying to deploy a SLES15 SP1 test SES6 cluster inside my company’s network. All the nodes are registered directly to SUSE.
They are VMs that are NAT’d on my laptop.

Access to SUSE is controlled by a proxy, and I have to set proxy values in /etc/sysconfig/proxy and I have set http_proxy/https_proxy in /etc/environment.

When try to this from inside the company, it always fails stage 3:

Retrieving repository ‘SLE-Module-Server-Applications15-SP1-Updates’ metadata [.error]
2020-01-30 10:54:37,944 [salt.loaded.int.module.cmdmod:830 ][ERROR ][70562] stderr: Repository ‘SLE-Module-Basesystem15-SP1-Pool’ is invalid.
[Basesystem_Module_15_SP1_x86_64:SLE-Module-Basesystem15-SP1-Pool|https://updates.suse.com/SUSE/Products/SLE-Module-Basesystem/15-SP1/x86_64/product?eUAhMu5_LPYFqK-RjEYRlVN-66EDDl46n9NEcpXe5
bcnX9IXed6Be_v7ucNepOmgvY9Bf8HtuTDBfOzyYXknY-SCBG1cYPvfnOFEDywMi6fwUqLlvJ_itUAEAYbXu64lKoYbbY6PnKsfUu_qvjuZJFaWEPuN2Per] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository ‘SLE-Module-Basesystem15-SP1-Pool’ because of the above error.
Repository ‘SLE-Module-Basesystem15-SP1-Updates’ is invalid.

However, I successfully do SUSEConnect -s and zypper commands interactively on each node. The failure is seen only with salt.

I can simulate this by doing:

ses6-admin:/var/cache/salt/master # salt ‘ses6-admin.ceph’ cmd.run ‘SUSEConnect -s’
ses6-admin.ceph:
SUSEConnect error: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3/TLS write client hello: tlsv1 alert access denied
ERROR: Minions returned with non-zero exit code

Connecting to the same node with ssh works fine:

ses6-admin:/var/cache/salt/master # ssh ses6-admin SUSEConnect -s
The authenticity of host ‘ses6-admin (192.168.122.10)’ can’t be established.
ECDSA key fingerprint is SHA256:/zLnsvxpqZN9w+fdzDB6XPi4aKMAezJgXn1p/zkwYXg.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘ses6-admin’ (ECDSA) to the list of known hosts.
[{“identifier”:“SLES”,“version”:“15.1”,“arch”:“x86_64”,“status”:“Registered”,“regcode”:“6A0E8047A63998”,“starts_at”:“2016-07-28 00:00:00 UTC”,“expires_at”:“2021-07-28 00:00:00 UTC”,“subscription_status”:“ACTIVE”,“type”:“full”},{“identifier”:“sle-module-basesystem”,“version”:“15.1”,“arch”:“x86_64”,“status”:“Registered”},{“identifier”:“sle-module-server-applications”,“version”:“15.1”,“arch”:“x86_64”,“status”:“Registered”},{“identifier”:“ses”,“version”:“6”,“arch”:“x86_64”,“status”:“Registered”,“regcode”:“E7F2A11D207B30”,“starts_at”:“2016-07-28 00:00:00 UTC”,“expires_at”:“2021-07-28 00:00:00 UTC”,“subscription_status”:“ACTIVE”,“type”:“full”}]

We can also see that the salt minion does not source /etc/environment:

ses6-admin:/var/cache/salt/master # salt ‘ses6-admin.ceph’ cmd.run ‘env’
ses6-admin.ceph:
LC_MEASUREMENT=C
LC_PAPER=C
LC_MONETARY=C
LANG=en_US.UTF-8
INVOCATION_ID=69003f2bf1d940658b9441d4e3cb5a17
NOTIFY_SOCKET=/run/systemd/notify
LC_NAME=C
LC_COLLATE=C
PWD=/root
LC_CTYPE=C
JOURNAL_STREAM=9:20413
LC_ADDRESS=C
LC_NUMERIC=C
LC_MESSAGES=C
SHLVL=1
LANGUAGE=C
LC_TELEPHONE=C
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LC_IDENTIFICATION=C
LC_TIME=C
_=/usr/bin/env

This all seems to be coming from deepsea:

ses6-admin:/var/cache/salt/master # rpm -qf /srv/salt/_modules/packagemanager.py
deepsea-0.9.27+git.0.93a84d2ea-3.9.1.noarch

Is there a way with deepsea/salt to do one of:

a - force use of the environment variables within salt.
b- tell deepsea/salt to ignore trying to update
c- have zypper resolve the values in /etc/sysconfig/proxy as we would expect for an interactive session?

Thanks,

Rick

[jschmid1]

Hey Rick,

thanks for your question.

First of all I’d like to ask you to verify if packagemanager.py is not working properly by executing:

1. (on the node that you would like to patch)

salt-call -l debug packagemanager.patch

salt-call -l debug packagemanager.up

2. (on the master node)

salt -l debug \$target_ndoe packagemanager.up

salt -l debug \$target_ndoe packagemanager.patch

Thanks

[Rick]

Apologies in advance for bad formatting. I’ll do several responses since one would be very long.

From one of the minions:

From minion:

Proxy is set in /etc/sysconfig/proxy

PROXY_ENABLED=“yes”
HTTP_PROXY=“http://web-proxy.corp.xxx.yyy:8080”
HTTPS_PROXY=“http://web-proxy.corp.xxx.yyy:8080”
FTP_PROXY=""
GOPHER_PROXY=""
SOCKS_PROXY=""
NO_PROXY=“localhost, 127.0.0.1”

Until I define http_proxy/https_proxy, SUSEConnect fails:

ses6-osd1:~ # SUSEConnect -s
SUSEConnect error: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3/TLS write client hello: tlsv1 alert access denied
ses6-osd1:~ # . /etc/environment
ses6-osd1:~ # SUSEConnect -s
[{“identifier”:“SLES”,“version”:“15.1”,“arch”:“x86_64”,“status”:“Registered”,“regcode”:“6A0E8047A63998”,“starts_at”:“2016-07-28 00:00:00 UTC”,“expires_at”:“2021-07-28 00:00:00 UTC”,“subscription_status”:“ACTIVE”,“type”:“full”},{“identifier”:“sle-module-basesystem”,“version”:“15.1”,“arch”:“x86_64”,“status”:“Registered”},{“identifier”:“sle-module-server-applications”,“version”:“15.1”,“arch”:“x86_64”,“status”:“Registered”},{“identifier”:“ses”,“version”:“6”,“arch”:“x86_64”,“status”:“Registered”,“regcode”:“E7F2A11D207B30”,“starts_at”:“2016-07-28 00:00:00 UTC”,“expires_at”:“2021-07-28 00:00:00 UTC”,“subscription_status”:“ACTIVE”,“type”:“full”}]

Ok, so try patch
ses6-osd1:~ # salt-call -l debug packagemanager.patch
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Including configuration from ‘/etc/salt/minion.d/_schedule.conf’
[DEBUG ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG ] Including configuration from ‘/etc/salt/minion.d/crc_method.conf’
[DEBUG ] Reading configuration from /etc/salt/minion.d/crc_method.conf
[DEBUG ] Including configuration from ‘/etc/salt/minion.d/master.conf’
[DEBUG ] Reading configuration from /etc/salt/minion.d/master.conf
[DEBUG ] Including configuration from ‘/etc/salt/minion.d/mine_functions.conf’
[DEBUG ] Reading configuration from /etc/salt/minion.d/mine_functions.conf
[DEBUG ] Guessing ID. The id can be explicitly set in /etc/salt/minion
[DEBUG ] Found minion id from generate_minion_id(): ses6-osd1.ceph
[DEBUG ] Error while parsing IPv4 address: ses6-osd1.ceph
[DEBUG ] Expected 4 octets in ‘ses6-osd1.ceph’
[DEBUG ] Error while parsing IPv6 address: ses6-osd1.ceph
[DEBUG ] At least 3 parts expected in ‘ses6-osd1.ceph’
[DEBUG ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] Grains refresh requested. Refreshing grains.
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Including configuration from ‘/etc/salt/minion.d/_schedule.conf’
[DEBUG ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG ] Including configuration from ‘/etc/salt/minion.d/crc_method.conf’
[DEBUG ] Reading configuration from /etc/salt/minion.d/crc_method.conf
[DEBUG ] Including configuration from ‘/etc/salt/minion.d/master.conf’
[DEBUG ] Reading configuration from /etc/salt/minion.d/master.conf
[DEBUG ] Including configuration from ‘/etc/salt/minion.d/mine_functions.conf’
[DEBUG ] Reading configuration from /etc/salt/minion.d/mine_functions.conf
[DEBUG ] Error while parsing IPv4 address: ::1
[DEBUG ] Expected 4 octets in ‘::1’
[DEBUG ] Error while parsing IPv4 address: fe80::5054:ff:feb6:d6bd
[DEBUG ] Expected 4 octets in ‘fe80::5054:ff:feb6:d6bd’
[DEBUG ] Unable to resolve address fe80::5054:ff:feb6:d6bd: [Errno 1] Unknown host
[DEBUG ] Elapsed time getting FQDNs: 0.11042904853820801 seconds
[DEBUG ] Error while parsing IPv4 address: ::1
[DEBUG ] Expected 4 octets in ‘::1’
[DEBUG ] Error while parsing IPv4 address: fe80::5054:ff:feb6:d6bd
[DEBUG ] Expected 4 octets in ‘fe80::5054:ff:feb6:d6bd’
[DEBUG ] Loading static grains from /etc/salt/grains
[DEBUG ] LazyLoaded zfs.is_supported
[DEBUG ] Connecting to master. Attempt 1 of 1
[DEBUG ] Error while parsing IPv4 address: ses6-admin
[DEBUG ] Expected 4 octets in ‘ses6-admin’
[DEBUG ] Error while parsing IPv6 address: ses6-admin
[DEBUG ] At least 3 parts expected in ‘ses6-admin’
[DEBUG ] “ses6-admin” Not an IP address? Assuming it is a hostname.
[DEBUG ] Master URI: tcp://192.168.122.10:4506
[DEBUG ] Initializing new AsyncAuth for (’/etc/salt/pki/minion’, ‘ses6-osd1.ceph’, ‘tcp://192.168.122.10:4506’)
[DEBUG ] Generated random reconnect delay between ‘1000ms’ and ‘11000ms’ (6846)
[DEBUG ] Setting zmq_reconnect_ivl to ‘6846ms’
[DEBUG ] Setting zmq_reconnect_ivl_max to ‘11000ms’
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (’/etc/salt/pki/minion’, ‘ses6-osd1.ceph’, ‘tcp://192.168.122.10:4506’, ‘clear’)
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.122.10:4506
[DEBUG ] Trying to connect to: tcp://192.168.122.10:4506
[DEBUG ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG ] Decrypting the current master AES key
[DEBUG ] salt.crypt.get_rsa_key: Loading private key
[DEBUG ] salt.crypt._get_key_with_evict: Loading private key
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] Connecting the Minion to the Master publish port, using the URI: tcp://192.168.122.10:4505
[DEBUG ] salt.crypt.get_rsa_key: Loading private key
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG ] Determining pillar cache
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (’/etc/salt/pki/minion’, ‘ses6-osd1.ceph’, ‘tcp://192.168.122.10:4506’, ‘aes’)
[DEBUG ] Initializing new AsyncAuth for (’/etc/salt/pki/minion’, ‘ses6-osd1.ceph’, ‘tcp://192.168.122.10:4506’)
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.122.10:4506
[DEBUG ] Trying to connect to: tcp://192.168.122.10:4506
[DEBUG ] salt.crypt.get_rsa_key: Loading private key
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] LazyLoaded packagemanager.patch
[DEBUG ] LazyLoaded direct_call.execute
[INFO ] Found suse. Using Zypper
[DEBUG ] Executing [‘zypper’, ‘–non-interactive’, ‘patch-check’] <<< this is what zypper is trying to do
[INFO ] No Patches Needed
[INFO ] No updates available.
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (’/etc/salt/pki/minion’, ‘ses6-osd1.ceph’, ‘tcp://192.168.122.10:4506’, ‘aes’)
[DEBUG ] Initializing new AsyncAuth for (’/etc/salt/pki/minion’, ‘ses6-osd1.ceph’, ‘tcp://192.168.122.10:4506’)
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.122.10:4506
[DEBUG ] Trying to connect to: tcp://192.168.122.10:4506
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] LazyLoaded nested.output
local:
None

This seems wrong because if I execute the zypper command that patch was trying to do, I get a different answer:

ses6-osd1:~ # zypper --non-interactive patch-check
Refreshing service ‘Basesystem_Module_15_SP1_x86_64’.
Refreshing service ‘SUSE_Enterprise_Storage_6_x86_64’.
Refreshing service ‘SUSE_Linux_Enterprise_Server_15_SP1_x86_64’.
Refreshing service ‘Server_Applications_Module_15_SP1_x86_64’.
Loading repository data…
Reading installed packages…
[B]
Found 6 applicable patches:
Category | Patches
------------±-------
security | 5
recommended | 1

6 patches needed (5 security patches)[/B]

Now for the update test:

ses6-osd1:~ # salt-call -l debug packagemanager.up
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Including configuration from ‘/etc/salt/minion.d/_schedule.conf’
[DEBUG ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG ] Including configuration from ‘/etc/salt/minion.d/crc_method.conf’
[DEBUG ] Reading configuration from /etc/salt/minion.d/crc_method.conf
[DEBUG ] Including configuration from ‘/etc/salt/minion.d/master.conf’
[DEBUG ] Reading configuration from /etc/salt/minion.d/master.conf
[DEBUG ] Including configuration from ‘/etc/salt/minion.d/mine_functions.conf’
[DEBUG ] Reading configuration from /etc/salt/minion.d/mine_functions.conf
[DEBUG ] Guessing ID. The id can be explicitly set in /etc/salt/minion
[DEBUG ] Found minion id from generate_minion_id(): ses6-osd1.ceph
[DEBUG ] Error while parsing IPv4 address: ses6-osd1.ceph
[DEBUG ] Expected 4 octets in ‘ses6-osd1.ceph’
[DEBUG ] Error while parsing IPv6 address: ses6-osd1.ceph
[DEBUG ] At least 3 parts expected in ‘ses6-osd1.ceph’
[DEBUG ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] Grains refresh requested. Refreshing grains.
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Including configuration from ‘/etc/salt/minion.d/_schedule.conf’
[DEBUG ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG ] Including configuration from ‘/etc/salt/minion.d/crc_method.conf’
[DEBUG ] Reading configuration from /etc/salt/minion.d/crc_method.conf
[DEBUG ] Including configuration from ‘/etc/salt/minion.d/master.conf’
[DEBUG ] Reading configuration from /etc/salt/minion.d/master.conf
[DEBUG ] Including configuration from ‘/etc/salt/minion.d/mine_functions.conf’
[DEBUG ] Reading configuration from /etc/salt/minion.d/mine_functions.conf
[DEBUG ] Error while parsing IPv4 address: ::1
[DEBUG ] Expected 4 octets in ‘::1’
[DEBUG ] Error while parsing IPv4 address: fe80::5054:ff:feb6:d6bd
[DEBUG ] Expected 4 octets in ‘fe80::5054:ff:feb6:d6bd’
[DEBUG ] Unable to resolve address fe80::5054:ff:feb6:d6bd: [Errno 1] Unknown host
[DEBUG ] Elapsed time getting FQDNs: 0.11236405372619629 seconds
[DEBUG ] Error while parsing IPv4 address: ::1
[DEBUG ] Expected 4 octets in ‘::1’
[DEBUG ] Error while parsing IPv4 address: fe80::5054:ff:feb6:d6bd
[DEBUG ] Expected 4 octets in ‘fe80::5054:ff:feb6:d6bd’
[DEBUG ] Loading static grains from /etc/salt/grains
[DEBUG ] LazyLoaded zfs.is_supported
[DEBUG ] Connecting to master. Attempt 1 of 1
[DEBUG ] Error while parsing IPv4 address: ses6-admin
[DEBUG ] Expected 4 octets in ‘ses6-admin’
[DEBUG ] Error while parsing IPv6 address: ses6-admin
[DEBUG ] At least 3 parts expected in ‘ses6-admin’
[DEBUG ] “ses6-admin” Not an IP address? Assuming it is a hostname.
[DEBUG ] Master URI: tcp://192.168.122.10:4506
[DEBUG ] Initializing new AsyncAuth for (’/etc/salt/pki/minion’, ‘ses6-osd1.ceph’, ‘tcp://192.168.122.10:4506’)
[DEBUG ] Generated random reconnect delay between ‘1000ms’ and ‘11000ms’ (10238)
[DEBUG ] Setting zmq_reconnect_ivl to ‘10238ms’
[DEBUG ] Setting zmq_reconnect_ivl_max to ‘11000ms’
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (’/etc/salt/pki/minion’, ‘ses6-osd1.ceph’, ‘tcp://192.168.122.10:4506’, ‘clear’)
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.122.10:4506
[DEBUG ] Trying to connect to: tcp://192.168.122.10:4506
[DEBUG ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG ] Decrypting the current master AES key
[DEBUG ] salt.crypt.get_rsa_key: Loading private key
[DEBUG ] salt.crypt._get_key_with_evict: Loading private key
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] Connecting the Minion to the Master publish port, using the URI: tcp://192.168.122.10:4505
[DEBUG ] salt.crypt.get_rsa_key: Loading private key
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG ] Determining pillar cache
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (’/etc/salt/pki/minion’, ‘ses6-osd1.ceph’, ‘tcp://192.168.122.10:4506’, ‘aes’)
[DEBUG ] Initializing new AsyncAuth for (’/etc/salt/pki/minion’, ‘ses6-osd1.ceph’, ‘tcp://192.168.122.10:4506’)
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.122.10:4506
[DEBUG ] Trying to connect to: tcp://192.168.122.10:4506
[DEBUG ] salt.crypt.get_rsa_key: Loading private key
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] LazyLoaded packagemanager.up
[DEBUG ] LazyLoaded direct_call.execute
[INFO ] Found suse. Using Zypper
[INFO ] Refreshing Repositories…
[ERROR ] Refreshing failed. There might be an issue resolving the repos
[DEBUG ] Executed [‘zypper’, ‘–non-interactive’, ‘refresh’]
[DEBUG ] Executing zypper lu | grep -sq ‘No updates found’
[INFO ] Update Needed
[DEBUG ] Executing [‘zypper’, ‘–non-interactive’, ‘up’, ‘–replacefiles’, ‘–auto-agree-with-licenses’]
[INFO ] returncode for [‘zypper’, ‘–non-interactive’, ‘up’, ‘–replacefiles’, ‘–auto-agree-with-licenses’] : 0
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (’/etc/salt/pki/minion’, ‘ses6-osd1.ceph’, ‘tcp://192.168.122.10:4506’, ‘aes’)
[DEBUG ] Initializing new AsyncAuth for (’/etc/salt/pki/minion’, ‘ses6-osd1.ceph’, ‘tcp://192.168.122.10:4506’)
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.122.10:4506
[DEBUG ] Trying to connect to: tcp://192.168.122.10:4506
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] LazyLoaded nested.output
local:
None

At least here we agree.

ses6-osd1:~ # zypper up
Refreshing service ‘Basesystem_Module_15_SP1_x86_64’.
Refreshing service ‘SUSE_Enterprise_Storage_6_x86_64’.
Refreshing service ‘SUSE_Linux_Enterprise_Server_15_SP1_x86_64’.
Refreshing service ‘Server_Applications_Module_15_SP1_x86_64’.
Loading repository data…
Reading installed packages…

Nothing to do.

[Rick]

Now from the master node:

e
ses6-admin:~ # salt -l debug ‘ses6-osd1.ceph’ packagemanager.up
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from ‘/etc/salt/master.d/crc_method.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/crc_method.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/eauth.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/eauth.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/logging.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/logging.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/modules.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/modules.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/output.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/output.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/reactor.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/salt-api.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/sharedsecret.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/sharedsecret.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: ses6-admin.ceph
[DEBUG ] Reading configuration from /root/.saltrc
[DEBUG ] Configuration file path: /root/.saltrc
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from ‘/etc/salt/master.d/crc_method.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/crc_method.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/eauth.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/eauth.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/logging.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/logging.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/modules.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/modules.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/output.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/output.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/reactor.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/salt-api.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/sharedsecret.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/sharedsecret.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: ses6-admin.ceph
[DEBUG ] Reading configuration from /root/.saltrc
[DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (’/etc/salt/pki/master’, ‘ses6-admin.ceph_master’, ‘tcp://127.0.0.1:4506’, ‘clear’)
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] LazyLoaded local_cache.get_load
[DEBUG ] Reading minion list from /var/cache/salt/master/jobs/f4/6fea52cd30cd915c54527c05c3f29f5210a534a1afaf16d85f9e522a8c6c32/.minions.p
[DEBUG ] get_iter_returns for jid 20200131114145136258 sent to {‘ses6-osd1.ceph’} will timeout at 11:41:50.145174
[DEBUG ] Checking whether jid 20200131114145136258 is still running
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (’/etc/salt/pki/master’, ‘ses6-admin.ceph_master’, ‘tcp://127.0.0.1:4506’, ‘clear’)
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] retcode missing from client return
[DEBUG ] jid 20200131114145136258 return from ses6-osd1.ceph
[DEBUG ] return event: {‘ses6-osd1.ceph’: {‘ret’: None, ‘retcode’: 0, ‘jid’: ‘20200131114145136258’}}
[DEBUG ] LazyLoaded nested.output
ses6-osd1.ceph:
None
[DEBUG ] jid 20200131114145136258 found all minions {‘ses6-osd1.ceph’}
[DEBUG ] Closing IPCMessageSubscriber instance

ses6-admin:~ # salt -l debug ‘ses6-osd1.ceph’ packagemanager.patch
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from ‘/etc/salt/master.d/crc_method.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/crc_method.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/eauth.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/eauth.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/logging.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/logging.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/modules.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/modules.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/output.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/output.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/reactor.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/salt-api.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/sharedsecret.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/sharedsecret.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: ses6-admin.ceph
[DEBUG ] Reading configuration from /root/.saltrc
[DEBUG ] Configuration file path: /root/.saltrc
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from ‘/etc/salt/master.d/crc_method.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/crc_method.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/eauth.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/eauth.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/logging.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/logging.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/modules.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/modules.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/output.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/output.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/reactor.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/salt-api.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG ] Including configuration from ‘/etc/salt/master.d/sharedsecret.conf’
[DEBUG ] Reading configuration from /etc/salt/master.d/sharedsecret.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: ses6-admin.ceph
[DEBUG ] Reading configuration from /root/.saltrc
[DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (’/etc/salt/pki/master’, ‘ses6-admin.ceph_master’, ‘tcp://127.0.0.1:4506’, ‘clear’)
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] LazyLoaded local_cache.get_load
[DEBUG ] Reading minion list from /var/cache/salt/master/jobs/1a/7085b53703ac856f1255c52de47f2b488f7a66326f355d2cdc502ed810d38b/.minions.p
[DEBUG ] get_iter_returns for jid 20200131114304170310 sent to {‘ses6-osd1.ceph’} will timeout at 11:43:09.177018
[DEBUG ] jid 20200131114304170310 return from ses6-osd1.ceph
[DEBUG ] return event: {‘ses6-osd1.ceph’: {‘ret’: None, ‘retcode’: 0, ‘jid’: ‘20200131114304170310’}}
[DEBUG ] LazyLoaded nested.output
ses6-osd1.ceph:
None
[DEBUG ] jid 20200131114304170310 found all minions {‘ses6-osd1.ceph’}
[DEBUG ] Closing IPCMessageSubscriber instance

[Rick]

I admit I might be making assumptions about what is failing by focusing on packagemanager but I based it on trying to get through stage 2:

es6-admin:~ # salt-run state.orch ceph.stage.2
deepsea_minions : valid
yaml_syntax : valid
public network : 192.168.122.0/24
cluster network : 192.168.122.0/24
[ERROR ] {‘out’: ‘highstate’, ‘ret’: {‘ses6-osd2.ceph’: {‘pkg_|-ceph_|-ceph_|-installed’: {‘result’: False, ‘name’: ‘ceph’, ‘changes’: {}, ‘comment’: ‘An exception occurred in this state: Traceback (most recent call last):
File “/usr/lib/python3.6/site-packages/salt/state.py”, line 1939, in call
ret = self.states[cdata[\‘full\’]](*cdata[\‘args\’], **cdata[\‘kwargs\’ ])
File “/usr/lib/python3.6/site-packages/salt/loader.py”, line 1982, in wrapper
return f(*args, **kwargs)
File “/usr/lib/python3.6/site-packages/salt/states/pkg.py”, line 1542, in installed
pkgs, refresh = _resolve_capabilities(pkgs, refresh=refresh, **kwargs)
File “/usr/lib/python3.6/site-packages/salt/states/pkg.py”, line 949, in _resolve_capabilities
ret = salt[\‘pkg.resolve_capabilities\’](pkgs, refresh=refresh, **kwargs)
File “/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py”, line 2793, in resolve_capabilities
refresh_db(root)
File “/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py”, line 1298, in refresh_db
out = zypper(root=root).refreshable.call(\‘refresh\’, \’–force\’)
File “/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py”, line 351, in _call
raise CommandExecutionError(\‘Zypper command failure: {0}\’.format(self.error_msg))
salt.exc eptions.CommandExecutionError: Zypper command failure: Repository \‘SLE-Module-Basesystem15-SP1-Pool\’ is invalid.
[Basesystem_Module_15_SP1_x86_64:SLE-Module-Basesystem15-SP1-Pool|https:/ /updates.suse.com/SUSE/Products/SLE-Module-Basesystem/15-SP1/x86_64/product?kBthW4isTKxZevHd9nldkzSn85pQo6CylmjPI3yiueWJJhnezED3Xuq6PLpuNyW2ca4pfDtIryp04zoH-7ViW-W_ePowEr0dxBIgiz2yevuMtuMkR GVQtMllJqaQmm057n-zifdMZSl4-v9eatj-MqzBT4xUbU4p] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Skippin g repository \‘SLE-Module-Basesystem15-SP1-Pool\’ because of the above error.
Repository \‘SLE-Module-Basesystem15-SP1-Updates\’ is invalid.
[Basesystem_Module_15_SP1_x86_64:SLE-Module-Ba sesystem15-SP1-Updates|https://updates.suse.com/SUSE/Updates/SLE-Module-Basesystem/15-SP1/x86_64/update?JCtyRwgXZeHNEYQLEGz9fjlycl_Nhg58wE_q96EJvWPpN-DBYQxkvnozmcWNB8KdrOA6eOo9MTruljI-5az1L q_Cn6P_9TSaTGiigjbfuuURJjHgClIzpuRptIqjiZBZXxtKUBJ5Vkac8rK1lWSWhOtaisIE8w] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository \‘SLE-Module-Basesystem15-SP1-Updates\’ because of the above error.
Repository \‘SUSE-Enterprise-Storage-6-Pool\’ is invalid.
[SUSE_Enterprise_Stora ge_6_x86_64:SUSE-Enterprise-Storage-6-Pool|https://updates.suse.com/SUSE/Products/Storage/6/x86_64/product?Bis3NMAbS0RVKCyYkXgKxdRDs6sPkb2r8WkwUqmLzfCvOT7RB4P-wyHb67UESJlTu94q2Y5iU_Rmm7H8fg oRX-AKKlxmPUknJgoSjwtydnyA476XAoaNjEj3fSrXUbBzY9yHT00] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
S kipping repository \‘SUSE-Enterprise-Storage-6-Pool\’ because of the above error.
Repository \‘SUSE-Enterprise-Storage-6-Updates\’ is invalid.
[SUSE_Enterprise_Storage_6_x86_64:SUSE-Enter prise-Storage-6-Updates|https://updates.suse.com/SUSE/Updates/Storage/6/x86_64/update?aIhStPdsLHeTgi6H8kTYQKaVZ-625HIZUsuvhjjzjAbFnc4iLsDrNu6nHZDDZcNzR9LEPOeAPovZtwJGLsZ9OS4uhYRFLkqv2QoesKi N5STR4g7tt33jt0IGFpO7DgUq3LhJ] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository \‘SUS E-Enterprise-Storage-6-Updates\’ because of the above error.
Repository \‘SLE-Product-SLES15-SP1-Pool\’ is invalid.
[SUSE_Linux_Enterprise_Server_15_SP1_x86_64:SLE-Product-SLES15-SP1-Pool |[url]https://updates.suse.com/SUSE/Products/SLE-Product-SLES/15-SP1/x86_64/product?351dbB-rik7-Jlc7UQem5dUL5qa_PfgXiHncmgx4Dl51K3tiGBy3LwplgwYXG1iiK0UJgGBz5b9qibVvyp06wECgP-WXoYeNjVXMtbI4V-p3x[/url] fyvQYd4lewSzJtPDY3J3th1F95XEhLDVub3TzHyZWykw] Valid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping r epository \‘SLE-Product-SLES15-SP1-Pool\’ because of the above error.
Repository \‘SLE-Product-SLES15-SP1-Updates\’ is invalid.\

…

later in the output

ses6-admin.ceph_master:
Name: push.proposal - Function: salt.runner - Result: Changed Started: - 09:36:16.254802 Duration: 480.942 ms
Name: refresh_pillar1 - Function: salt.state - Result: Changed Started: - 09:36:16.735915 Duration: 1272.131 ms
Name: advise.networks - Function: salt.runner - Result: Changed Started: - 09:36:18.008243 Duration: 989.363 ms

       ID: install ceph packages
 Function: salt.state
   Result: False
  Comment: Run failed on minions: ses6-osd3.ceph, ses6-osd1.ceph, ses6-mon3.ceph, ses6-admin.ceph, ses6-mds.ceph, ses6-osd2.ceph, ses6-mon1.ceph, ses6-mon2.ceph
  Started: 09:36:18.998271
 Duration: 20332.113 ms
  Changes:
           ses6-osd2.ceph:
           ----------
                     ID: ceph
               Function: pkg.installed
                 Result: False
                Comment: An exception occurred in this state: Traceback (most recent call last):
                           File "/usr/lib/python3.6/site-packages/salt/state.py", line 1939, in call
                             ret = self.states[cdata['full']](*cdata['args'], **cdata['kwargs'])
                           File "/usr/lib/python3.6/site-packages/salt/loader.py", line 1982, in wrapper
                             return f(*args, **kwargs)
                           File "/usr/lib/python3.6/site-packages/salt/states/pkg.py", line 1542, in installed
                             pkgs, refresh = _resolve_capabilities(pkgs, refresh=refresh, **kwargs)
                           File "/usr/lib/python3.6/site-packages/salt/states/pkg.py", line 949, in _resolve_capabilities
                             ret = __salt__['pkg.resolve_capabilities'](pkgs, refresh=refresh, **kwargs)
                           File "/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py", line 2793, in resolve_capabilities
                             refresh_db(root)
                           File "/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py", line 1298, in refresh_db
                             out = __zypper__(root=root).refreshable.call('refresh', '--force')
                           File "/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py", line 351, in __call
                             raise CommandExecutionError('Zypper command failure: {0}'.format(self.error_msg))
                         salt.exceptions.CommandExecutionError: Zypper command failure: Repository 'SLE-Module-Basesystem15-SP1-Pool' is invalid.
                         [Basesystem_Module_15_SP1_x86_64:SLE-Module-Basesystem15-SP1-Pool|[url]https://updates.suse.com/SUSE/Products/SLE-Module-Basesystem/15-SP1/x86_64/product?kBthW4isTKxZ[/url] evHd9nldkzSn85pQo6CylmjPI3yiueWJJhnezED3Xuq6PLpuNyW2ca4pfDtIryp04zoH-7ViW-W_ePowEr0dxBIgiz2yevuMtuMkRGVQtMllJqaQmm057n-zifdMZSl4-v9eatj-MqzBT4xUbU4p] Valid metadata not found at specified U RL
                         Please check if the URIs defined for this repository are pointing to a valid repository.
                         Skipping repository 'SLE-Module-Basesystem15-SP1-Pool' because of the above error.
                         Repository 'SLE-Module-Basesystem15-SP1-Updates' is invalid.

… and this continues for each repo

1/x86_64/update?V50pwYUUdR72OOsSe2Verb5HcFGxUQfFrgqmeUfK2okdX4aufiTmqA046jESvp9G46m0_14oq6z4ots2OznxSUCDfXDES6qj5-BUiXZUOaVkPZmO03TmtR_9FGaQI2x4ZsiHjiYVsWMFmRtMouEsiT_nrPHP9Eld_OBJu9DjNw] V alid metadata not found at specified URL
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository ‘SLE-Module-Server-Applications15-SP1-Updates’ because of the above error.
Could not refresh the repositories because of errors.
Started: 09:36:35.809102
Duration: 3484.176 ms
Changes:

           Summary for ses6-admin.ceph
           ------------
           Succeeded: 0
           Failed:    1
           ------------
           Total states run:     1
           Total run time:   3.484 s

Summary for ses6-admin.ceph_master

Succeeded: 3 (changed=3)
Failed: 1

Total states run: 4
Total run time: 23.075 s

[a_jaeger]

I suggest to open a ticket with SUSE’s support to dig deeper into this.

[Rick]

@a_jaeger;59355

I suggest to open a ticket with SUSE’s support to dig deeper into this.

Thanks. I’ve worked around it after a fashion by manually doing updates on the master and all the minions. This gets me past stage 1 and I finally have a working environment. Thanks for pointing out the diagnostic method with -l debug as that was revealing.