Can't add Hosts (Health TimeOut)

Rancher 1.x
1

Suddenly when adding new host it dosent properly add to rancher.
Health Agent fails with a time out and not much log info from other containers. Nothing really changes just stopped working.

Component Version
Rancher v1.4.3
Cattle v0.176.11
User Interface v1.4.9
Rancher CLI v0.4.1
Rancher Compose v0.12.2

Network Manager has this errors
7/9/2017 2:43:42 PMtime=“2017-07-09T20:43:42Z” level=info msg="Starting event router."
7/9/2017 2:43:42 PMiptables: No chain/target/match by that name.
7/9/2017 2:43:42 PMtime=“2017-07-09T20:43:42Z” level=info msg="Container [795984be201e6fe320528f0161d31dfb3d8e8a3b27c379de079d025d0f892da8] not running. Can’t setup resolv.conf."
7/9/2017 2:43:45 PMtime=“2017-07-09T20:43:45Z” level=info msg="Setting up resolv.conf for ContainerId [cc6a8a71f975573bfb2633456b971b048b2226319036a156a93aebccde16f529]"
7/9/2017 2:43:45 PMtime=“2017-07-09T20:43:45Z” level=info msg=“CNI up” cid=cc6a8a71f975573bfb2633456b971b048b2226319036a156a93aebccde16f529 networkMode=ipsec
7/9/2017 2:43:45 PMtime=“2017-07-09T20:43:45Z” level=info msg=“CNI up done” cid=cc6a8a71f975573bfb2633456b971b048b2226319036a156a93aebccde16f529 networkMode=ipsec result=
7/9/2017 2:44:49 PMtime=“2017-07-09T20:44:49Z” level=info msg=“CNI down” cid=cc6a8a71f975573bfb2633456b971b048b2226319036a156a93aebccde16f529 networkMode=ipsec
7/9/2017 2:45:26 PMtime=“2017-07-09T20:45:26Z” level=info msg="Setting up resolv.conf for ContainerId [610a4dcedf72c8451e4a051e6af22a05224f6e2e31067e7d0147a45583aeae2b]"
7/9/2017 2:45:26 PMtime=“2017-07-09T20:45:26Z” level=info msg=“CNI up” cid=610a4dcedf72c8451e4a051e6af22a05224f6e2e31067e7d0147a45583aeae2b networkMode=ipsec
7/9/2017 2:45:26 PMtime=“2017-07-09T20:45:26Z” level=info msg=“CNI up done” cid=610a4dcedf72c8451e4a051e6af22a05224f6e2e31067e7d0147a45583aeae2b networkMode=ipsec result=
7/9/2017 2:46:30 PMtime=“2017-07-09T20:46:30Z” level=info msg=“CNI down” cid=610a4dcedf72c8451e4a051e6af22a05224f6e2e31067e7d0147a45583aeae2b networkMode=ipsec
7/9/2017 2:47:11 PMtime=“2017-07-09T20:47:11Z” level=info msg="Setting up resolv.conf for ContainerId [b0a82eeb2d933e053caba27cfac8c03b35e60e3d35927d1b4ee8ff3c76613997]"
7/9/2017 2:47:11 PMtime=“2017-07-09T20:47:11Z” level=info msg=“CNI up” cid=b0a82eeb2d933e053caba27cfac8c03b35e60e3d35927d1b4ee8ff3c76613997 networkMode=ipsec
7/9/2017 2:47:11 PMtime=“2017-07-09T20:47:11Z” level=info msg=“CNI up done” cid=b0a82eeb2d933e053caba27cfac8c03b35e60e3d35927d1b4ee8ff3c76613997 networkMode=ipsec result=
7/9/2017 2:48:15 PMtime=“2017-07-09T20:48:15Z” level=info msg=“CNI down” cid=b0a82eeb2d933e053caba27cfac8c03b35e60e3d35927d1b4ee8ff3c76613997 networkMode=ipsec
7/9/2017 2:49:11 PMnet.ipv4.conf.docker0.route_localnet = 1
7/9/2017 2:49:11 PMiptables: No chain/target/match by that name.
7/9/2017 2:49:11 PMtime=“2017-07-09T20:49:11Z” level=info msg="Setting up resolv.conf for ContainerId [2aee2b241ae8be0c9a4e4467130aa25582dbbce02005704ce2cb545d14a4fc61]"
7/9/2017 2:49:11 PMtime=“2017-07-09T20:49:11Z” level=info msg=“CNI up” cid=2aee2b241ae8be0c9a4e4467130aa25582dbbce02005704ce2cb545d14a4fc61 networkMode=ipsec
7/9/2017 2:49:11 PMtime=“2017-07-09T20:49:11Z” level=info msg=“CNI up done” cid=2aee2b241ae8be0c9a4e4467130aa25582dbbce02005704ce2cb545d14a4fc61 networkMode=ipsec result=

I am willing to upgrade but afraid it will not add the other host when upgrade. ipsec has the status Updating-Active (In Progress) all the time.

Any thoughts, I checked make sure we have ports open etc

2

Can you share the logs of the rancher-agent container on the new host.

3

time=“2017-07-10T14:39:08Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=a00a5d64-1805-48c0-b4d3-b1025d43c7b9 eventName=compute.instance.activate resourceId=1ihm127677
time=“2017-07-10T14:39:08Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: d95e3682-3729-43dc-8a32-5374cbee7c23, Resource Id: 1ihm127677”
time=“2017-07-10T14:39:08Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=d95e3682-3729-43dc-8a32-5374cbee7c23 eventName=compute.instance.activate resourceId=1ihm127677
time=“2017-07-10T14:39:08Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: d538574a-8d38-4b03-b3f6-737e41ffd262, Resource Id: 1ihm127677”
time=“2017-07-10T14:39:40Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=d538574a-8d38-4b03-b3f6-737e41ffd262 eventName=compute.instance.activate resourceId=1ihm127677
time=“2017-07-10T14:39:40Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: b004fb02-4f1f-474b-9c59-ba23394c27a9, Resource Id: 1ihm127677”
time=“2017-07-10T14:39:40Z” level=info msg=“rancher id [1230077]: Container with docker id [425e0f838dd4f12f23d2fb95d16bec2aff892030f4adcd03a77764807ef72b66] has been deactivated”
time=“2017-07-10T14:39:40Z” level=info msg=“Reply: b004fb02-4f1f-474b-9c59-ba23394c27a9, compute.instance.deactivate, 1ihm127677:instanceHostMap”
time=“2017-07-10T14:41:32Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: 566e9151-c960-4f1b-9e14-d3b19bef9954, Resource Id: 1vspm257356”
time=“2017-07-10T14:41:32Z” level=info msg=“rancher id [1vspm257356]: Volume with name has been removed”
time=“2017-07-10T14:41:32Z” level=info msg=“Reply: 566e9151-c960-4f1b-9e14-d3b19bef9954, storage.volume.remove, 1vspm257356:volumeStoragePoolMap”
time=“2017-07-10T14:43:50Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: 62a8149d-f044-430f-910d-fa0de21580b8, Resource Id: 1ispm125821”
time=“2017-07-10T14:43:50Z” level=info msg=“Reply: 62a8149d-f044-430f-910d-fa0de21580b8, storage.image.activate, 1ispm125821:imageStoragePoolMap”
time=“2017-07-10T14:43:50Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: df76187c-d678-406a-9574-f7303821a0c9, Resource Id: 1vspm257357”
time=“2017-07-10T14:43:50Z” level=info msg=“Reply: df76187c-d678-406a-9574-f7303821a0c9, storage.volume.activate, 1vspm257357:volumeStoragePoolMap”
time=“2017-07-10T14:43:51Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: fd229d63-a319-4f23-82e5-61730995f1a0, Resource Id: 1ihm127678”
time=“2017-07-10T14:43:51Z” level=info msg=“rancher id [1230078]: Container with docker id [fcedf0bb5c9b567a2f52906a2e99eb4a4d9f125d49660b4aa6dc6990ab00867e] has been started”
time=“2017-07-10T14:44:23Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=fd229d63-a319-4f23-82e5-61730995f1a0 eventName=compute.instance.activate resourceId=1ihm127678
time=“2017-07-10T14:44:23Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 7fdab0b3-7bbd-41ba-971b-a394b226172f, Resource Id: 1ihm127678”
time=“2017-07-10T14:44:23Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=7fdab0b3-7bbd-41ba-971b-a394b226172f eventName=compute.instance.activate resourceId=1ihm127678
time=“2017-07-10T14:44:23Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: ebd6f534-8355-40d1-8ff3-348de9ae57f8, Resource Id: 1ihm127678”
time=“2017-07-10T14:44:55Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=ebd6f534-8355-40d1-8ff3-348de9ae57f8 eventName=compute.instance.activate resourceId=1ihm127678
time=“2017-07-10T14:44:55Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: e82b4294-5394-40e1-9d17-3868359fe3d8, Resource Id: 1ihm127678”
time=“2017-07-10T14:44:55Z” level=info msg=“rancher id [1230078]: Container with docker id [fcedf0bb5c9b567a2f52906a2e99eb4a4d9f125d49660b4aa6dc6990ab00867e] has been deactivated”
time=“2017-07-10T14:44:55Z” level=info msg=“Reply: e82b4294-5394-40e1-9d17-3868359fe3d8, compute.instance.deactivate, 1ihm127678:instanceHostMap”
time=“2017-07-10T14:46:16Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: b935cbdd-8244-4346-ba55-db508b9cc1c7, Resource Id: 1ihm127676”
time=“2017-07-10T14:46:16Z” level=info msg=“Reply: b935cbdd-8244-4346-ba55-db508b9cc1c7, compute.instance.remove, 1ihm127676:instanceHostMap”
time=“2017-07-10T14:46:46Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: d8dd4de6-47e4-409e-9f4b-2af83404635e, Resource Id: 1vspm257357”
time=“2017-07-10T14:46:46Z” level=info msg=“rancher id [1vspm257357]: Volume with name has been removed”
time=“2017-07-10T14:46:46Z” level=info msg=“Reply: d8dd4de6-47e4-409e-9f4b-2af83404635e, storage.volume.remove, 1vspm257357:volumeStoragePoolMap”
time=“2017-07-10T14:48:16Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: 0fb61b0f-895d-4c77-a6ce-4c1de41937a0, Resource Id: 1ihm127677”
time=“2017-07-10T14:48:16Z” level=info msg=“Reply: 0fb61b0f-895d-4c77-a6ce-4c1de41937a0, compute.instance.remove, 1ihm127677:instanceHostMap”
time=“2017-07-10T14:49:04Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: 4fe30642-8f6c-4662-a99c-11b85601cd74, Resource Id: 1ispm125822”
time=“2017-07-10T14:49:04Z” level=info msg=“Reply: 4fe30642-8f6c-4662-a99c-11b85601cd74, storage.image.activate, 1ispm125822:imageStoragePoolMap”
time=“2017-07-10T14:49:04Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: ec854dc2-d958-4d63-9319-18bdd3c96808, Resource Id: 1vspm257358”
time=“2017-07-10T14:49:04Z” level=info msg=“Reply: ec854dc2-d958-4d63-9319-18bdd3c96808, storage.volume.activate, 1vspm257358:volumeStoragePoolMap”
time=“2017-07-10T14:49:05Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 6466cb8e-c3eb-492e-97be-2f7bf5f5bb00, Resource Id: 1ihm127679”
time=“2017-07-10T14:49:05Z” level=info msg=“rancher id [1230079]: Container with docker id [23ffbe3c4ce2757b557ddbe5f5b89445c46f8de59e514b5c101d93791a0d486f] has been started”
time=“2017-07-10T14:49:37Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=6466cb8e-c3eb-492e-97be-2f7bf5f5bb00 eventName=compute.instance.activate resourceId=1ihm127679
time=“2017-07-10T14:49:37Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 68ed105b-580f-40e8-b482-fbbd48b57685, Resource Id: 1ihm127679”
time=“2017-07-10T14:49:37Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=68ed105b-580f-40e8-b482-fbbd48b57685 eventName=compute.instance.activate resourceId=1ihm127679
time=“2017-07-10T14:49:37Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 854f688e-1069-4779-b6a0-c01893bc25e9, Resource Id: 1ihm127679”
time=“2017-07-10T14:50:09Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=854f688e-1069-4779-b6a0-c01893bc25e9 eventName=compute.instance.activate resourceId=1ihm127679
time=“2017-07-10T14:50:09Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: 2433d45b-85f8-4f7b-8eb1-c58c70bfb363, Resource Id: 1ihm127679”
time=“2017-07-10T14:50:09Z” level=info msg=“rancher id [1230079]: Container with docker id [23ffbe3c4ce2757b557ddbe5f5b89445c46f8de59e514b5c101d93791a0d486f] has been deactivated”
time=“2017-07-10T14:50:09Z” level=info msg=“Reply: 2433d45b-85f8-4f7b-8eb1-c58c70bfb363, compute.instance.deactivate, 1ihm127679:instanceHostMap”
time=“2017-07-10T14:52:01Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: bfe7863b-57df-41fd-9eba-1d0bc0953523, Resource Id: 1vspm257358”
time=“2017-07-10T14:52:01Z” level=info msg=“rancher id [1vspm257358]: Volume with name has been removed”
time=“2017-07-10T14:52:01Z” level=info msg=“Reply: bfe7863b-57df-41fd-9eba-1d0bc0953523, storage.volume.remove, 1vspm257358:volumeStoragePoolMap”
time=“2017-07-10T14:54:16Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: 6504afba-47cb-43d8-95b6-f717ecce9e90, Resource Id: 1ihm127679”
time=“2017-07-10T14:54:16Z” level=info msg=“Reply: 6504afba-47cb-43d8-95b6-f717ecce9e90, compute.instance.remove, 1ihm127679:instanceHostMap”
time=“2017-07-10T14:54:19Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: 098832e8-95c5-4ccc-8dfe-98ccabdd4e2e, Resource Id: 1ispm125823”
time=“2017-07-10T14:54:19Z” level=info msg=“Reply: 098832e8-95c5-4ccc-8dfe-98ccabdd4e2e, storage.image.activate, 1ispm125823:imageStoragePoolMap”
time=“2017-07-10T14:54:19Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: 370ca6ac-1f4b-4086-a56b-997f05ebd374, Resource Id: 1vspm257359”
time=“2017-07-10T14:54:19Z” level=info msg=“Reply: 370ca6ac-1f4b-4086-a56b-997f05ebd374, storage.volume.activate, 1vspm257359:volumeStoragePoolMap”
time=“2017-07-10T14:54:19Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 8eba265f-f12e-4dba-9560-db5d445b54ef, Resource Id: 1ihm127680”
time=“2017-07-10T14:54:20Z” level=info msg=“rancher id [1230080]: Container with docker id [f0e012ee91f89675b82ccb9cb3fa3f31de7529a88123ec2d55d82cb819a00a15] has been started”
time=“2017-07-10T14:54:51Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=8eba265f-f12e-4dba-9560-db5d445b54ef eventName=compute.instance.activate resourceId=1ihm127680
time=“2017-07-10T14:54:52Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 02314e33-33de-4851-b1ed-4f5f3a40cff0, Resource Id: 1ihm127680”
time=“2017-07-10T14:54:52Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=02314e33-33de-4851-b1ed-4f5f3a40cff0 eventName=compute.instance.activate resourceId=1ihm127680
time=“2017-07-10T14:54:52Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: f31fd498-068f-49dc-a7c7-772250b29fd0, Resource Id: 1ihm127680”
time=“2017-07-10T14:55:16Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: 07cb53de-1e64-477d-82bf-bc254ce8e838, Resource Id: 1ihm127678”
time=“2017-07-10T14:55:16Z” level=info msg=“Reply: 07cb53de-1e64-477d-82bf-bc254ce8e838, compute.instance.remove, 1ihm127678:instanceHostMap”
time=“2017-07-10T14:55:23Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=f31fd498-068f-49dc-a7c7-772250b29fd0 eventName=compute.instance.activate resourceId=1ihm127680
time=“2017-07-10T14:55:24Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: a7862dd0-7522-4d19-b053-b6abfec37885, Resource Id: 1ihm127680”
time=“2017-07-10T14:55:24Z” level=info msg=“rancher id [1230080]: Container with docker id [f0e012ee91f89675b82ccb9cb3fa3f31de7529a88123ec2d55d82cb819a00a15] has been deactivated”
time=“2017-07-10T14:55:24Z” level=info msg=“Reply: a7862dd0-7522-4d19-b053-b6abfec37885, compute.instance.deactivate, 1ihm127680:instanceHostMap”
time=“2017-07-10T14:57:17Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: 388dbeb3-414c-4599-b950-74607e728e98, Resource Id: 1vspm257359”
time=“2017-07-10T14:57:17Z” level=info msg=“rancher id [1vspm257359]: Volume with name has been removed”
time=“2017-07-10T14:57:17Z” level=info msg=“Reply: 388dbeb3-414c-4599-b950-74607e728e98, storage.volume.remove, 1vspm257359:volumeStoragePoolMap”
time=“2017-07-10T14:59:35Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: ca1e09a5-a390-4597-a170-1b5dd5ab8614, Resource Id: 1ispm125824”
time=“2017-07-10T14:59:35Z” level=info msg=“Reply: ca1e09a5-a390-4597-a170-1b5dd5ab8614, storage.image.activate, 1ispm125824:imageStoragePoolMap”
time=“2017-07-10T14:59:35Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: e36a0d4e-6f54-40ff-b4aa-e139f51bcb9b, Resource Id: 1vspm257360”
time=“2017-07-10T14:59:35Z” level=info msg=“Reply: e36a0d4e-6f54-40ff-b4aa-e139f51bcb9b, storage.volume.activate, 1vspm257360:volumeStoragePoolMap”
time=“2017-07-10T14:59:35Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 37370ac7-da7f-4a0a-a708-3c87553732f2, Resource Id: 1ihm127681”
time=“2017-07-10T14:59:36Z” level=info msg=“rancher id [1230081]: Container with docker id [45add9fb2b78630eefef223a34461a1fc979342672061264cdb5eac6926dcf79] has been started”
time=“2017-07-10T15:00:07Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=37370ac7-da7f-4a0a-a708-3c87553732f2 eventName=compute.instance.activate resourceId=1ihm127681
time=“2017-07-10T15:00:07Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 90cb3baf-5ee4-4631-a888-eb7d683af9fc, Resource Id: 1ihm127681”
time=“2017-07-10T15:00:08Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=90cb3baf-5ee4-4631-a888-eb7d683af9fc eventName=compute.instance.activate resourceId=1ihm127681
time=“2017-07-10T15:00:08Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 5ef36e92-0ae4-4b55-be44-6948daecd8ef, Resource Id: 1ihm127681”
time=“2017-07-10T15:00:39Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=5ef36e92-0ae4-4b55-be44-6948daecd8ef eventName=compute.instance.activate resourceId=1ihm127681
time=“2017-07-10T15:00:40Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: 3cfbef0b-c44a-4ea8-b7ae-93014096b85e, Resource Id: 1ihm127681”
time=“2017-07-10T15:00:40Z” level=info msg=“rancher id [1230081]: Container with docker id [45add9fb2b78630eefef223a34461a1fc979342672061264cdb5eac6926dcf79] has been deactivated”
time=“2017-07-10T15:00:40Z” level=info msg=“Reply: 3cfbef0b-c44a-4ea8-b7ae-93014096b85e, compute.instance.deactivate, 1ihm127681:instanceHostMap”
time=“2017-07-10T15:02:31Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: 4c94e40a-22d6-4706-a79c-5d33e7aeb163, Resource Id: 1vspm257360”
time=“2017-07-10T15:02:31Z” level=info msg=“rancher id [1vspm257360]: Volume with name has been removed”
time=“2017-07-10T15:02:31Z” level=info msg=“Reply: 4c94e40a-22d6-4706-a79c-5d33e7aeb163, storage.volume.remove, 1vspm257360:volumeStoragePoolMap”
time=“2017-07-10T15:04:49Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: 7e920d4e-61e6-4044-8bfa-afb8a80a9fd7, Resource Id: 1ispm125825”
time=“2017-07-10T15:04:49Z” level=info msg=“Reply: 7e920d4e-61e6-4044-8bfa-afb8a80a9fd7, storage.image.activate, 1ispm125825:imageStoragePoolMap”
time=“2017-07-10T15:04:49Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: f3ca6d07-63a7-4c12-a5c2-b83def490b5c, Resource Id: 1vspm257361”
time=“2017-07-10T15:04:49Z” level=info msg=“Reply: f3ca6d07-63a7-4c12-a5c2-b83def490b5c, storage.volume.activate, 1vspm257361:volumeStoragePoolMap”
time=“2017-07-10T15:04:49Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 70fabae0-7f4a-427e-98b0-574038f3d302, Resource Id: 1ihm127682”
time=“2017-07-10T15:04:49Z” level=info msg=“rancher id [1230082]: Container with docker id [da9fcaa23e92ab5fc85c326befe53823c9a227f9bc430d31e246cdb03d6ebe32] has been started”
time=“2017-07-10T15:05:21Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=70fabae0-7f4a-427e-98b0-574038f3d302 eventName=compute.instance.activate resourceId=1ihm127682
time=“2017-07-10T15:05:21Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 55860bc0-aa7d-4a19-a191-4df92a0803d9, Resource Id: 1ihm127682”
time=“2017-07-10T15:05:21Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=55860bc0-aa7d-4a19-a191-4df92a0803d9 eventName=compute.instance.activate resourceId=1ihm127682
time=“2017-07-10T15:05:21Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 7cfe41a9-4e7d-41ab-8c84-683300fed869, Resource Id: 1ihm127682”
time=“2017-07-10T15:05:53Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=7cfe41a9-4e7d-41ab-8c84-683300fed869 eventName=compute.instance.activate resourceId=1ihm127682
time=“2017-07-10T15:05:53Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: ed1aa9cc-937b-48b0-b579-9648e9e0dc84, Resource Id: 1ihm127682”
time=“2017-07-10T15:05:53Z” level=info msg=“rancher id [1230082]: Container with docker id [da9fcaa23e92ab5fc85c326befe53823c9a227f9bc430d31e246cdb03d6ebe32] has been deactivated”
time=“2017-07-10T15:05:53Z” level=info msg=“Reply: ed1aa9cc-937b-48b0-b579-9648e9e0dc84, compute.instance.deactivate, 1ihm127682:instanceHostMap”
time=“2017-07-10T15:06:16Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: c151bc71-1887-488f-bbad-7cc119915a75, Resource Id: 1ihm127680”
time=“2017-07-10T15:06:16Z” level=info msg=“Reply: c151bc71-1887-488f-bbad-7cc119915a75, compute.instance.remove, 1ihm127680:instanceHostMap”
time=“2017-07-10T15:07:46Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: d8a8804f-869c-4bc1-bd03-5f15e8e581b3, Resource Id: 1vspm257361”
time=“2017-07-10T15:07:46Z” level=info msg=“rancher id [1vspm257361]: Volume with name has been removed”
time=“2017-07-10T15:07:46Z” level=info msg=“Reply: d8a8804f-869c-4bc1-bd03-5f15e8e581b3, storage.volume.remove, 1vspm257361:volumeStoragePoolMap”
time=“2017-07-10T15:09:16Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: d0ef135a-156b-4c9e-ab78-2ccc93ad3b12, Resource Id: 1ihm127681”
time=“2017-07-10T15:09:16Z” level=info msg=“Reply: d0ef135a-156b-4c9e-ab78-2ccc93ad3b12, compute.instance.remove, 1ihm127681:instanceHostMap”
time=“2017-07-10T15:09:16Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: cc6ecb85-b5d6-4c3e-bf72-d8b50efce978, Resource Id: 1ihm127682”
time=“2017-07-10T15:09:16Z” level=info msg=“Reply: cc6ecb85-b5d6-4c3e-bf72-d8b50efce978, compute.instance.remove, 1ihm127682:instanceHostMap”
time=“2017-07-10T15:10:04Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: 144ca979-7a8a-4522-97e8-2bb86a0b6f61, Resource Id: 1ispm125826”
time=“2017-07-10T15:10:04Z” level=info msg=“Reply: 144ca979-7a8a-4522-97e8-2bb86a0b6f61, storage.image.activate, 1ispm125826:imageStoragePoolMap”
time=“2017-07-10T15:10:04Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: 1ebfb913-c444-47e6-890a-a2df3e3d691a, Resource Id: 1vspm257362”
time=“2017-07-10T15:10:04Z” level=info msg=“Reply: 1ebfb913-c444-47e6-890a-a2df3e3d691a, storage.volume.activate, 1vspm257362:volumeStoragePoolMap”
time=“2017-07-10T15:10:05Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 585be014-9e2d-41c5-a750-3e16ce3d7e01, Resource Id: 1ihm127683”
time=“2017-07-10T15:10:05Z” level=info msg=“rancher id [1230083]: Container with docker id [16d5976bf73c4e629c8828099f639102fe2a40c219b19777585637b8b539c829] has been started”
time=“2017-07-10T15:10:36Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=585be014-9e2d-41c5-a750-3e16ce3d7e01 eventName=compute.instance.activate resourceId=1ihm127683
time=“2017-07-10T15:10:37Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: ca43c488-4754-4771-b599-c97617805b19, Resource Id: 1ihm127683”
time=“2017-07-10T15:10:37Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=ca43c488-4754-4771-b599-c97617805b19 eventName=compute.instance.activate resourceId=1ihm127683
time=“2017-07-10T15:10:37Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: e04f5b24-413b-4d60-ae30-8ca236a2a9a2, Resource Id: 1ihm127683”
time=“2017-07-10T15:11:08Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=e04f5b24-413b-4d60-ae30-8ca236a2a9a2 eventName=compute.instance.activate resourceId=1ihm127683
time=“2017-07-10T15:11:09Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: bf32b780-fe97-41ee-bab3-cb052260dfc9, Resource Id: 1ihm127683”
time=“2017-07-10T15:11:09Z” level=info msg=“rancher id [1230083]: Container with docker id [16d5976bf73c4e629c8828099f639102fe2a40c219b19777585637b8b539c829] has been deactivated”
time=“2017-07-10T15:11:09Z” level=info msg=“Reply: bf32b780-fe97-41ee-bab3-cb052260dfc9, compute.instance.deactivate, 1ihm127683:instanceHostMap”
time=“2017-07-10T15:13:01Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: f1efde06-ec85-4829-9fb6-bba9dd347d18, Resource Id: 1vspm257362”
time=“2017-07-10T15:13:01Z” level=info msg=“rancher id [1vspm257362]: Volume with name has been removed”
time=“2017-07-10T15:13:01Z” level=info msg=“Reply: f1efde06-ec85-4829-9fb6-bba9dd347d18, storage.volume.remove, 1vspm257362:volumeStoragePoolMap”
time=“2017-07-10T15:15:19Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: c2faf876-dcce-497f-abba-9787a6d21a9d, Resource Id: 1ispm125827”
time=“2017-07-10T15:15:19Z” level=info msg=“Reply: c2faf876-dcce-497f-abba-9787a6d21a9d, storage.image.activate, 1ispm125827:imageStoragePoolMap”
time=“2017-07-10T15:15:19Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: 12fe7376-f9de-460b-af53-3a4bd53458f4, Resource Id: 1vspm257363”
time=“2017-07-10T15:15:19Z” level=info msg=“Reply: 12fe7376-f9de-460b-af53-3a4bd53458f4, storage.volume.activate, 1vspm257363:volumeStoragePoolMap”
time=“2017-07-10T15:15:20Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 667198e9-8d07-4b11-9656-38bddc1f0b03, Resource Id: 1ihm127684”
time=“2017-07-10T15:15:20Z” level=info msg=“rancher id [1230084]: Container with docker id [1560a36b46ff8b2a28175d11286a925d016502cd2644e4564f94d85e7b11f333] has been started”
time=“2017-07-10T15:15:52Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=667198e9-8d07-4b11-9656-38bddc1f0b03 eventName=compute.instance.activate resourceId=1ihm127684
time=“2017-07-10T15:15:52Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 97f349da-8b60-4c7a-963f-7f4e54a39001, Resource Id: 1ihm127684”
time=“2017-07-10T15:15:52Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=97f349da-8b60-4c7a-963f-7f4e54a39001 eventName=compute.instance.activate resourceId=1ihm127684
time=“2017-07-10T15:15:52Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: d30b77cd-4403-4fd3-83a4-22f3c95eb2ca, Resource Id: 1ihm127684”
time=“2017-07-10T15:16:24Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=d30b77cd-4403-4fd3-83a4-22f3c95eb2ca eventName=compute.instance.activate resourceId=1ihm127684
time=“2017-07-10T15:16:24Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: 2b556507-d5bf-4a9c-a402-79624d7cf40c, Resource Id: 1ihm127684”
time=“2017-07-10T15:16:24Z” level=info msg=“rancher id [1230084]: Container with docker id [1560a36b46ff8b2a28175d11286a925d016502cd2644e4564f94d85e7b11f333] has been deactivated”
time=“2017-07-10T15:16:24Z” level=info msg=“Reply: 2b556507-d5bf-4a9c-a402-79624d7cf40c, compute.instance.deactivate, 1ihm127684:instanceHostMap”
time=“2017-07-10T15:17:16Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: 6e460dc4-0cd8-46fd-af72-8480011e71c4, Resource Id: 1ihm127683”
time=“2017-07-10T15:17:16Z” level=info msg=“Reply: 6e460dc4-0cd8-46fd-af72-8480011e71c4, compute.instance.remove, 1ihm127683:instanceHostMap”
time=“2017-07-10T15:18:16Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: fb922cf4-1992-473d-bc9f-5d88e507edd6, Resource Id: 1vspm257363”
time=“2017-07-10T15:18:16Z” level=info msg=“rancher id [1vspm257363]: Volume with name has been removed”
time=“2017-07-10T15:18:16Z” level=info msg=“Reply: fb922cf4-1992-473d-bc9f-5d88e507edd6, storage.volume.remove, 1vspm257363:volumeStoragePoolMap”
time=“2017-07-10T15:20:36Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: 66f37a59-7293-43d7-821f-aff12acbbaf4, Resource Id: 1ispm125828”
time=“2017-07-10T15:20:36Z” level=info msg=“Reply: 66f37a59-7293-43d7-821f-aff12acbbaf4, storage.image.activate, 1ispm125828:imageStoragePoolMap”
time=“2017-07-10T15:20:36Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: 49b3bc80-0589-437b-8054-17d2ad8bc6e6, Resource Id: 1vspm257364”
time=“2017-07-10T15:20:36Z” level=info msg=“Reply: 49b3bc80-0589-437b-8054-17d2ad8bc6e6, storage.volume.activate, 1vspm257364:volumeStoragePoolMap”
time=“2017-07-10T15:20:36Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 2a62f265-f6ab-4eb8-a7a4-bad81f82ff91, Resource Id: 1ihm127685”
time=“2017-07-10T15:20:36Z” level=info msg=“rancher id [1230085]: Container with docker id [7cc215c9d8f7351313d916fb6bc3f47b6bb6dcc7eb2e575f7592fd803d6b6465] has been started”
time=“2017-07-10T15:21:08Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=2a62f265-f6ab-4eb8-a7a4-bad81f82ff91 eventName=compute.instance.activate resourceId=1ihm127685
time=“2017-07-10T15:21:08Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 7ebd9e96-3192-41d0-a7c0-725ca39211b1, Resource Id: 1ihm127685”
time=“2017-07-10T15:21:08Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=7ebd9e96-3192-41d0-a7c0-725ca39211b1 eventName=compute.instance.activate resourceId=1ihm127685
time=“2017-07-10T15:21:08Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 2a4ddabe-2ebd-4b84-a7d1-e3b4cabaf36b, Resource Id: 1ihm127685”
time=“2017-07-10T15:21:40Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=2a4ddabe-2ebd-4b84-a7d1-e3b4cabaf36b eventName=compute.instance.activate resourceId=1ihm127685
time=“2017-07-10T15:21:40Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: d5fbd3bb-017c-4786-b7b0-3cfcb3df058f, Resource Id: 1ihm127685”
time=“2017-07-10T15:21:40Z” level=info msg=“rancher id [1230085]: Container with docker id [7cc215c9d8f7351313d916fb6bc3f47b6bb6dcc7eb2e575f7592fd803d6b6465] has been deactivated”
time=“2017-07-10T15:21:40Z” level=info msg=“Reply: d5fbd3bb-017c-4786-b7b0-3cfcb3df058f, compute.instance.deactivate, 1ihm127685:instanceHostMap”
time=“2017-07-10T15:23:31Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: 2a179183-5025-404a-812c-435c427469fd, Resource Id: 1vspm257364”
time=“2017-07-10T15:23:31Z” level=info msg=“rancher id [1vspm257364]: Volume with name has been removed”
time=“2017-07-10T15:23:31Z” level=info msg=“Reply: 2a179183-5025-404a-812c-435c427469fd, storage.volume.remove, 1vspm257364:volumeStoragePoolMap”
time=“2017-07-10T15:24:16Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: f07daadd-ed01-4eab-90a7-4b52ee20535d, Resource Id: 1ihm127685”
time=“2017-07-10T15:24:16Z” level=info msg=“Reply: f07daadd-ed01-4eab-90a7-4b52ee20535d, compute.instance.remove, 1ihm127685:instanceHostMap”
time=“2017-07-10T15:25:49Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: 99edf34a-8282-48c0-b199-08bd62588636, Resource Id: 1ispm125829”
time=“2017-07-10T15:25:49Z” level=info msg=“Reply: 99edf34a-8282-48c0-b199-08bd62588636, storage.image.activate, 1ispm125829:imageStoragePoolMap”
time=“2017-07-10T15:25:50Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: 2d4de513-60eb-474f-9b18-514ed1792fea, Resource Id: 1vspm257365”
time=“2017-07-10T15:25:50Z” level=info msg=“Reply: 2d4de513-60eb-474f-9b18-514ed1792fea, storage.volume.activate, 1vspm257365:volumeStoragePoolMap”
time=“2017-07-10T15:25:50Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 14ea2be4-0928-4c43-a06c-a3b53d7a0e6f, Resource Id: 1ihm127686”
time=“2017-07-10T15:25:50Z” level=info msg=“rancher id [1230086]: Container with docker id [527bfca9f9b3c801bdca4c1e07cfb66404f599a9a0de1f58e35fd021a241871d] has been started”
time=“2017-07-10T15:26:22Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=14ea2be4-0928-4c43-a06c-a3b53d7a0e6f eventName=compute.instance.activate resourceId=1ihm127686
time=“2017-07-10T15:26:22Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: dca0dbf2-0c3f-4203-910a-fa9bd5e494f5, Resource Id: 1ihm127686”
time=“2017-07-10T15:26:22Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=dca0dbf2-0c3f-4203-910a-fa9bd5e494f5 eventName=compute.instance.activate resourceId=1ihm127686
time=“2017-07-10T15:26:22Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 596fd1bb-b48c-4a59-903c-0f6fafdb381a, Resource Id: 1ihm127686”
time=“2017-07-10T15:26:54Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=596fd1bb-b48c-4a59-903c-0f6fafdb381a eventName=compute.instance.activate resourceId=1ihm127686
time=“2017-07-10T15:26:54Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: 63f27bea-b8f4-4fa3-ad33-ccc51776c952, Resource Id: 1ihm127686”
time=“2017-07-10T15:26:54Z” level=info msg=“rancher id [1230086]: Container with docker id [527bfca9f9b3c801bdca4c1e07cfb66404f599a9a0de1f58e35fd021a241871d] has been deactivated”
time=“2017-07-10T15:26:54Z” level=info msg=“Reply: 63f27bea-b8f4-4fa3-ad33-ccc51776c952, compute.instance.deactivate, 1ihm127686:instanceHostMap”
time=“2017-07-10T15:27:16Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: a9c6dac2-b34b-4bb0-9cd8-10ef8fa6cc71, Resource Id: 1ihm127684”
time=“2017-07-10T15:27:16Z” level=info msg=“Reply: a9c6dac2-b34b-4bb0-9cd8-10ef8fa6cc71, compute.instance.remove, 1ihm127684:instanceHostMap”
time=“2017-07-10T15:28:47Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: cd822b80-ff87-459d-a5a0-461b52c28145, Resource Id: 1vspm257365”
time=“2017-07-10T15:28:47Z” level=info msg=“rancher id [1vspm257365]: Volume with name has been removed”
time=“2017-07-10T15:28:47Z” level=info msg=“Reply: cd822b80-ff87-459d-a5a0-461b52c28145, storage.volume.remove, 1vspm257365:volumeStoragePoolMap”
time=“2017-07-10T15:31:05Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: de2a5f51-29cd-47a2-b8df-9364154b7c56, Resource Id: 1ispm125830”
time=“2017-07-10T15:31:05Z” level=info msg=“Reply: de2a5f51-29cd-47a2-b8df-9364154b7c56, storage.image.activate, 1ispm125830:imageStoragePoolMap”
time=“2017-07-10T15:31:05Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: e407aee6-b2a4-4d59-b359-26f29a7a808f, Resource Id: 1vspm257366”
time=“2017-07-10T15:31:05Z” level=info msg=“Reply: e407aee6-b2a4-4d59-b359-26f29a7a808f, storage.volume.activate, 1vspm257366:volumeStoragePoolMap”
time=“2017-07-10T15:31:05Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 540ac6d2-b780-4563-9bb1-816d2343babe, Resource Id: 1ihm127687”
time=“2017-07-10T15:31:06Z” level=info msg=“rancher id [1230087]: Container with docker id [3b08437e85047248a55668cb5aa6dd5cf97d6fff2a4711541216f519142f5976] has been started”
time=“2017-07-10T15:31:37Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=540ac6d2-b780-4563-9bb1-816d2343babe eventName=compute.instance.activate resourceId=1ihm127687
time=“2017-07-10T15:31:37Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 0fb20b84-3b86-4972-a331-2fb66f25c13f, Resource Id: 1ihm127687”
time=“2017-07-10T15:31:37Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=0fb20b84-3b86-4972-a331-2fb66f25c13f eventName=compute.instance.activate resourceId=1ihm127687
time=“2017-07-10T15:31:37Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 6c586db7-3926-426c-9fa5-bf1e19cf9b3a, Resource Id: 1ihm127687”
time=“2017-07-10T15:32:09Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=6c586db7-3926-426c-9fa5-bf1e19cf9b3a eventName=compute.instance.activate resourceId=1ihm127687
time=“2017-07-10T15:32:09Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: 70e5243f-635a-4d28-af93-139f8a7af89d, Resource Id: 1ihm127687”
time=“2017-07-10T15:32:10Z” level=info msg=“rancher id [1230087]: Container with docker id [3b08437e85047248a55668cb5aa6dd5cf97d6fff2a4711541216f519142f5976] has been deactivated”
time=“2017-07-10T15:32:10Z” level=info msg=“Reply: 70e5243f-635a-4d28-af93-139f8a7af89d, compute.instance.deactivate, 1ihm127687:instanceHostMap”
time=“2017-07-10T15:34:03Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: 28ef59d7-4034-4b6e-a12a-e2be703964e2, Resource Id: 1vspm257366”
time=“2017-07-10T15:34:03Z” level=info msg=“rancher id [1vspm257366]: Volume with name has been removed”
time=“2017-07-10T15:34:03Z” level=info msg=“Reply: 28ef59d7-4034-4b6e-a12a-e2be703964e2, storage.volume.remove, 1vspm257366:volumeStoragePoolMap”
time=“2017-07-10T15:35:16Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: 6f55c102-46e5-4e92-9554-b734d872022d, Resource Id: 1ihm127686”
time=“2017-07-10T15:35:16Z” level=info msg=“Reply: 6f55c102-46e5-4e92-9554-b734d872022d, compute.instance.remove, 1ihm127686:instanceHostMap”
time=“2017-07-10T15:36:16Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: 73c51e60-0bdc-423e-ba6c-0697a173f9d9, Resource Id: 1ihm127687”
time=“2017-07-10T15:36:16Z” level=info msg=“Reply: 73c51e60-0bdc-423e-ba6c-0697a173f9d9, compute.instance.remove, 1ihm127687:instanceHostMap”
time=“2017-07-10T15:36:20Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: 9ca3f881-73f1-4037-a768-c67497b25678, Resource Id: 1ispm125831”
time=“2017-07-10T15:36:20Z” level=info msg=“Reply: 9ca3f881-73f1-4037-a768-c67497b25678, storage.image.activate, 1ispm125831:imageStoragePoolMap”
time=“2017-07-10T15:36:20Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: 309fa47f-c05f-4f0e-93e4-1fd07a4abce9, Resource Id: 1vspm257367”
time=“2017-07-10T15:36:20Z” level=info msg=“Reply: 309fa47f-c05f-4f0e-93e4-1fd07a4abce9, storage.volume.activate, 1vspm257367:volumeStoragePoolMap”
time=“2017-07-10T15:36:20Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 1e2e1474-7fa1-48ca-a466-2bd1a577e626, Resource Id: 1ihm127688”
time=“2017-07-10T15:36:20Z” level=info msg=“rancher id [1230088]: Container with docker id [60c350ccd1b93623b7bebbcbf2d171275a4f095fe459b41bcca03b87a1d27723] has been started”
time=“2017-07-10T15:36:52Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=1e2e1474-7fa1-48ca-a466-2bd1a577e626 eventName=compute.instance.activate resourceId=1ihm127688
time=“2017-07-10T15:36:52Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 9963b943-68ee-4994-9351-c67e675df20f, Resource Id: 1ihm127688”
time=“2017-07-10T15:36:52Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=9963b943-68ee-4994-9351-c67e675df20f eventName=compute.instance.activate resourceId=1ihm127688
time=“2017-07-10T15:36:52Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: f175648b-4c64-455b-a987-020f2280370d, Resource Id: 1ihm127688”
time=“2017-07-10T15:37:24Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=f175648b-4c64-455b-a987-020f2280370d eventName=compute.instance.activate resourceId=1ihm127688
time=“2017-07-10T15:37:25Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: ddc0bfa3-a71e-478b-8ea0-552b47e6ce86, Resource Id: 1ihm127688”
time=“2017-07-10T15:37:25Z” level=info msg=“rancher id [1230088]: Container with docker id [60c350ccd1b93623b7bebbcbf2d171275a4f095fe459b41bcca03b87a1d27723] has been deactivated”
time=“2017-07-10T15:37:25Z” level=info msg=“Reply: ddc0bfa3-a71e-478b-8ea0-552b47e6ce86, compute.instance.deactivate, 1ihm127688:instanceHostMap”
time=“2017-07-10T15:39:17Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: 23576a8a-f976-46bd-aeaf-8516ce26113d, Resource Id: 1vspm257367”
time=“2017-07-10T15:39:17Z” level=info msg=“rancher id [1vspm257367]: Volume with name has been removed”
time=“2017-07-10T15:39:17Z” level=info msg=“Reply: 23576a8a-f976-46bd-aeaf-8516ce26113d, storage.volume.remove, 1vspm257367:volumeStoragePoolMap”
time=“2017-07-10T15:41:35Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: 17794ba4-ead8-4e21-a1ca-bba43e876983, Resource Id: 1ispm125832”
time=“2017-07-10T15:41:35Z” level=info msg=“Reply: 17794ba4-ead8-4e21-a1ca-bba43e876983, storage.image.activate, 1ispm125832:imageStoragePoolMap”
time=“2017-07-10T15:41:35Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: 16db6391-7c2f-41ce-9cbe-c44bc30d8c53, Resource Id: 1vspm257368”
time=“2017-07-10T15:41:35Z” level=info msg=“Reply: 16db6391-7c2f-41ce-9cbe-c44bc30d8c53, storage.volume.activate, 1vspm257368:volumeStoragePoolMap”
time=“2017-07-10T15:41:36Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 87c01927-1633-4e22-81dd-33b61fadfe45, Resource Id: 1ihm127689”
time=“2017-07-10T15:41:36Z” level=info msg=“rancher id [1230089]: Container with docker id [b1ff943afe87776652cebb80c2db752e3c71a48444ae624eed0295fd0160d176] has been started”
time=“2017-07-10T15:42:08Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=87c01927-1633-4e22-81dd-33b61fadfe45 eventName=compute.instance.activate resourceId=1ihm127689
time=“2017-07-10T15:42:08Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: a08277dc-a8fd-4844-98e5-00fc10436ba5, Resource Id: 1ihm127689”
time=“2017-07-10T15:42:08Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=a08277dc-a8fd-4844-98e5-00fc10436ba5 eventName=compute.instance.activate resourceId=1ihm127689
time=“2017-07-10T15:42:08Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 2b967205-24ea-4c98-868b-c1a5badc5bb5, Resource Id: 1ihm127689”
time=“2017-07-10T15:42:39Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=2b967205-24ea-4c98-868b-c1a5badc5bb5 eventName=compute.instance.activate resourceId=1ihm127689
time=“2017-07-10T15:42:40Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: 905201ae-eeb7-4acc-b26f-67c4fbe4b6a6, Resource Id: 1ihm127689”
time=“2017-07-10T15:42:40Z” level=info msg=“rancher id [1230089]: Container with docker id [b1ff943afe87776652cebb80c2db752e3c71a48444ae624eed0295fd0160d176] has been deactivated”
time=“2017-07-10T15:42:40Z” level=info msg=“Reply: 905201ae-eeb7-4acc-b26f-67c4fbe4b6a6, compute.instance.deactivate, 1ihm127689:instanceHostMap”
time=“2017-07-10T15:43:31Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: 84f482ee-9675-4463-9aaa-158ac450b1cc, Resource Id: 1ihm127688”
time=“2017-07-10T15:43:31Z” level=info msg=“Reply: 84f482ee-9675-4463-9aaa-158ac450b1cc, compute.instance.remove, 1ihm127688:instanceHostMap”
time=“2017-07-10T15:44:32Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: 8a943ae0-4bf4-4bd6-b7bc-dfcae2bcdd0f, Resource Id: 1vspm257368”
time=“2017-07-10T15:44:32Z” level=info msg=“rancher id [1vspm257368]: Volume with name has been removed”
time=“2017-07-10T15:44:32Z” level=info msg=“Reply: 8a943ae0-4bf4-4bd6-b7bc-dfcae2bcdd0f, storage.volume.remove, 1vspm257368:volumeStoragePoolMap”
time=“2017-07-10T15:46:50Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: c46e9a2d-bea3-4daf-948f-e747e5a0636d, Resource Id: 1ispm125833”
time=“2017-07-10T15:46:50Z” level=info msg=“Reply: c46e9a2d-bea3-4daf-948f-e747e5a0636d, storage.image.activate, 1ispm125833:imageStoragePoolMap”
time=“2017-07-10T15:46:50Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: e6086153-df0b-48b0-841d-c35c850e59bd, Resource Id: 1vspm257369”
time=“2017-07-10T15:46:50Z” level=info msg=“Reply: e6086153-df0b-48b0-841d-c35c850e59bd, storage.volume.activate, 1vspm257369:volumeStoragePoolMap”
time=“2017-07-10T15:46:50Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 8e5f3e2e-602f-4148-b8a0-248013d0f4fd, Resource Id: 1ihm127690”
time=“2017-07-10T15:46:51Z” level=info msg=“rancher id [1230090]: Container with docker id [31d3d562dd524729c4a0303497ee3b2cfb63bc7f3fb2ab3d13586da9ca16b701] has been started”
time=“2017-07-10T15:47:22Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=8e5f3e2e-602f-4148-b8a0-248013d0f4fd eventName=compute.instance.activate resourceId=1ihm127690
time=“2017-07-10T15:47:22Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: c7295595-dc37-4673-a918-93120d67c02c, Resource Id: 1ihm127690”
time=“2017-07-10T15:47:22Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=c7295595-dc37-4673-a918-93120d67c02c eventName=compute.instance.activate resourceId=1ihm127690
time=“2017-07-10T15:47:22Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 4ce87cdf-0494-4b63-91a7-0e39c3a19486, Resource Id: 1ihm127690”
time=“2017-07-10T15:47:54Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=4ce87cdf-0494-4b63-91a7-0e39c3a19486 eventName=compute.instance.activate resourceId=1ihm127690
time=“2017-07-10T15:47:54Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: 45a3e624-4331-4805-a4d7-9dcffb01d60a, Resource Id: 1ihm127690”
time=“2017-07-10T15:47:55Z” level=info msg=“rancher id [1230090]: Container with docker id [31d3d562dd524729c4a0303497ee3b2cfb63bc7f3fb2ab3d13586da9ca16b701] has been deactivated”
time=“2017-07-10T15:47:55Z” level=info msg=“Reply: 45a3e624-4331-4805-a4d7-9dcffb01d60a, compute.instance.deactivate, 1ihm127690:instanceHostMap”
time=“2017-07-10T15:49:47Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: 13ba15db-3fb6-4b7d-952b-6586cb2b0dee, Resource Id: 1vspm257369”
time=“2017-07-10T15:49:47Z” level=info msg=“rancher id [1vspm257369]: Volume with name has been removed”
time=“2017-07-10T15:49:47Z” level=info msg=“Reply: 13ba15db-3fb6-4b7d-952b-6586cb2b0dee, storage.volume.remove, 1vspm257369:volumeStoragePoolMap”
time=“2017-07-10T15:51:32Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: c01db596-5ff7-4d35-b2f1-15ac08033782, Resource Id: 1ihm127689”
time=“2017-07-10T15:51:32Z” level=info msg=“Reply: c01db596-5ff7-4d35-b2f1-15ac08033782, compute.instance.remove, 1ihm127689:instanceHostMap”
time=“2017-07-10T15:52:05Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: e1437b33-0466-4f24-9ea1-720f4b61877b, Resource Id: 1ispm125834”
time=“2017-07-10T15:52:05Z” level=info msg=“Reply: e1437b33-0466-4f24-9ea1-720f4b61877b, storage.image.activate, 1ispm125834:imageStoragePoolMap”
time=“2017-07-10T15:52:05Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: 7e7ae6bf-90dd-4044-8a06-06657d8ced9f, Resource Id: 1vspm257370”
time=“2017-07-10T15:52:05Z” level=info msg=“Reply: 7e7ae6bf-90dd-4044-8a06-06657d8ced9f, storage.volume.activate, 1vspm257370:volumeStoragePoolMap”
time=“2017-07-10T15:52:05Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: b9aae88c-5b5b-4a00-aaba-006cf5beafe1, Resource Id: 1ihm127691”
time=“2017-07-10T15:52:06Z” level=info msg=“rancher id [1230091]: Container with docker id [211a59b47ed0591070c8bb02beb27a16dab058b2f82acb6c7bdf39f57e07a838] has been started”
time=“2017-07-10T15:52:37Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=b9aae88c-5b5b-4a00-aaba-006cf5beafe1 eventName=compute.instance.activate resourceId=1ihm127691
time=“2017-07-10T15:52:37Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 5550b448-dde2-4295-b985-0c8122d85206, Resource Id: 1ihm127691”
time=“2017-07-10T15:52:37Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=5550b448-dde2-4295-b985-0c8122d85206 eventName=compute.instance.activate resourceId=1ihm127691
time=“2017-07-10T15:52:38Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: a82a6b19-e9ea-486d-91ce-d99deecc28ee, Resource Id: 1ihm127691”
time=“2017-07-10T15:53:09Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=a82a6b19-e9ea-486d-91ce-d99deecc28ee eventName=compute.instance.activate resourceId=1ihm127691
time=“2017-07-10T15:53:09Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: 588041be-368b-4b3b-99f6-51992298fe59, Resource Id: 1ihm127691”
time=“2017-07-10T15:53:10Z” level=info msg=“rancher id [1230091]: Container with docker id [211a59b47ed0591070c8bb02beb27a16dab058b2f82acb6c7bdf39f57e07a838] has been deactivated”
time=“2017-07-10T15:53:10Z” level=info msg=“Reply: 588041be-368b-4b3b-99f6-51992298fe59, compute.instance.deactivate, 1ihm127691:instanceHostMap”
time=“2017-07-10T15:55:03Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: 680f7b1f-e9db-4f83-8108-e452a4ad3eae, Resource Id: 1vspm257370”
time=“2017-07-10T15:55:03Z” level=info msg=“rancher id [1vspm257370]: Volume with name has been removed”
time=“2017-07-10T15:55:03Z” level=info msg=“Reply: 680f7b1f-e9db-4f83-8108-e452a4ad3eae, storage.volume.remove, 1vspm257370:volumeStoragePoolMap”
time=“2017-07-10T15:57:21Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: 2adb5c72-1984-4b40-9131-2f3a3b08729e, Resource Id: 1ispm125835”
time=“2017-07-10T15:57:21Z” level=info msg=“Reply: 2adb5c72-1984-4b40-9131-2f3a3b08729e, storage.image.activate, 1ispm125835:imageStoragePoolMap”
time=“2017-07-10T15:57:21Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: 04aa5e15-6dee-4c00-a8a4-71e8a9e567d7, Resource Id: 1vspm257371”
time=“2017-07-10T15:57:21Z” level=info msg=“Reply: 04aa5e15-6dee-4c00-a8a4-71e8a9e567d7, storage.volume.activate, 1vspm257371:volumeStoragePoolMap”
time=“2017-07-10T15:57:21Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 353f5dbd-3ea5-4604-8967-9d14857711ab, Resource Id: 1ihm127692”
time=“2017-07-10T15:57:22Z” level=info msg=“rancher id [1230092]: Container with docker id [b7831b6c48784b0c4609a70e9ef3c46235ea79e0fff6bf05508737a3e7a04b70] has been started”
time=“2017-07-10T15:57:53Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=353f5dbd-3ea5-4604-8967-9d14857711ab eventName=compute.instance.activate resourceId=1ihm127692
time=“2017-07-10T15:57:53Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: a80ac0a1-41c0-4d63-91f4-741a763f8b26, Resource Id: 1ihm127692”
time=“2017-07-10T15:57:53Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=a80ac0a1-41c0-4d63-91f4-741a763f8b26 eventName=compute.instance.activate resourceId=1ihm127692
time=“2017-07-10T15:57:54Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 3db4d606-bc41-49b9-8457-0ec1d34dee9b, Resource Id: 1ihm127692”
time=“2017-07-10T15:58:25Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=3db4d606-bc41-49b9-8457-0ec1d34dee9b eventName=compute.instance.activate resourceId=1ihm127692
time=“2017-07-10T15:58:26Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: 8052a9bf-b735-4c91-a1d4-f00e02ebe7e0, Resource Id: 1ihm127692”
time=“2017-07-10T15:58:26Z” level=info msg=“rancher id [1230092]: Container with docker id [b7831b6c48784b0c4609a70e9ef3c46235ea79e0fff6bf05508737a3e7a04b70] has been deactivated”
time=“2017-07-10T15:58:26Z” level=info msg=“Reply: 8052a9bf-b735-4c91-a1d4-f00e02ebe7e0, compute.instance.deactivate, 1ihm127692:instanceHostMap”
time=“2017-07-10T15:58:32Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: 44ef19c1-8a20-4676-a931-c9236c320991, Resource Id: 1ihm127690”
time=“2017-07-10T15:58:32Z” level=info msg=“Reply: 44ef19c1-8a20-4676-a931-c9236c320991, compute.instance.remove, 1ihm127690:instanceHostMap”
time=“2017-07-10T16:00:17Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: 1af3be5a-bd44-4ef5-8c81-3654716c2ca4, Resource Id: 1vspm257371”
time=“2017-07-10T16:00:17Z” level=info msg=“rancher id [1vspm257371]: Volume with name has been removed”
time=“2017-07-10T16:00:17Z” level=info msg=“Reply: 1af3be5a-bd44-4ef5-8c81-3654716c2ca4, storage.volume.remove, 1vspm257371:volumeStoragePoolMap”
time=“2017-07-10T16:02:32Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: 3dcb4f3c-de25-4c3a-9765-28c60f0755be, Resource Id: 1ihm127692”
time=“2017-07-10T16:02:32Z” level=info msg=“Reply: 3dcb4f3c-de25-4c3a-9765-28c60f0755be, compute.instance.remove, 1ihm127692:instanceHostMap”
time=“2017-07-10T16:02:32Z” level=info msg=“Received event: Name: compute.instance.remove, Event Id: 8f702db2-8420-4248-90d4-05d161593279, Resource Id: 1ihm127691”
time=“2017-07-10T16:02:32Z” level=info msg=“Reply: 8f702db2-8420-4248-90d4-05d161593279, compute.instance.remove, 1ihm127691:instanceHostMap”
time=“2017-07-10T16:02:34Z” level=info msg=“Received event: Name: storage.image.activate, Event Id: 5bd6db04-5847-4cb1-9c85-7642782c6797, Resource Id: 1ispm125836”
time=“2017-07-10T16:02:34Z” level=info msg=“Reply: 5bd6db04-5847-4cb1-9c85-7642782c6797, storage.image.activate, 1ispm125836:imageStoragePoolMap”
time=“2017-07-10T16:02:36Z” level=info msg=“Received event: Name: storage.volume.activate, Event Id: 3199a2a7-4709-4355-879a-5dc1ac7b0607, Resource Id: 1vspm257372”
time=“2017-07-10T16:02:36Z” level=info msg=“Reply: 3199a2a7-4709-4355-879a-5dc1ac7b0607, storage.volume.activate, 1vspm257372:volumeStoragePoolMap”
time=“2017-07-10T16:02:37Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 49cbd633-7efe-4eef-96d7-767c8ec00432, Resource Id: 1ihm127693”
time=“2017-07-10T16:02:37Z” level=info msg=“rancher id [1230093]: Container with docker id [fcc043f99d9de1430bd68062c398b320af93b0aa7a653d164380737d3e8fa229] has been started”
time=“2017-07-10T16:03:09Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=49cbd633-7efe-4eef-96d7-767c8ec00432 eventName=compute.instance.activate resourceId=1ihm127693
time=“2017-07-10T16:03:09Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: adfc8740-9856-47b2-8eb2-e0cbf344f145, Resource Id: 1ihm127693”
time=“2017-07-10T16:03:09Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=adfc8740-9856-47b2-8eb2-e0cbf344f145 eventName=compute.instance.activate resourceId=1ihm127693
time=“2017-07-10T16:03:09Z” level=info msg=“Received event: Name: compute.instance.activate, Event Id: 396fb637-e801-4d39-b149-35241182a654, Resource Id: 1ihm127693”
time=“2017-07-10T16:03:41Z” level=error msg=“Error processing event” err=Timeout getting IP address eventId=396fb637-e801-4d39-b149-35241182a654 eventName=compute.instance.activate resourceId=1ihm127693
time=“2017-07-10T16:03:41Z” level=info msg=“Received event: Name: compute.instance.deactivate, Event Id: 0ff7c093-ef90-4838-a64d-c208b86d39f5, Resource Id: 1ihm127693”
time=“2017-07-10T16:03:41Z” level=info msg=“rancher id [1230093]: Container with docker id [fcc043f99d9de1430bd68062c398b320af93b0aa7a653d164380737d3e8fa229] has been deactivated”
time=“2017-07-10T16:03:41Z” level=info msg=“Reply: 0ff7c093-ef90-4838-a64d-c208b86d39f5, compute.instance.deactivate, 1ihm127693:instanceHostMap”
time=“2017-07-10T16:05:32Z” level=info msg=“Received event: Name: storage.volume.remove, Event Id: 23a98df8-6e04-48d5-b8ac-4ced6659979c, Resource Id: 1vspm257372”
time=“2017-07-10T16:05:32Z” level=info msg=“rancher id [1vspm257372]: Volume with name has been removed”
time=“2017-07-10T16:05:32Z” level=info msg=“Reply: 23a98df8-6e04-48d5-b8ac-4ced6659979c, storage.volume.remove, 1vspm257372:volumeStoragePoolMap”

4

@rm2016 this is getting into bug and triage territory. Can you open up a github issue with these details instead? I can’t promise that someone will address it immediately, but it will be better manged there.

5

I did notice this error log on server over and over

2017-07-11 09:13:45,573 ERROR [:] [] [] [] [ecutorService-5] [i.c.p.agent.impl.AgentLocatorImpl ] Failed to find instance to delegate to for agent [15303] uri [delegate:///?vnetId=4742&networkServiceProviderId=1]
2017-07-11 09:13:45,574 ERROR [:] [] [] [] [ecutorService-5] [.p.c.v.i.ConfigItemStatusManagerImpl] Error updating item(s) [agent-instance-startup, healthcheck, iptables, hosts, ipsec-hosts, ipsec, node-services] on [agent:153
03] com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Delegate [delegate:///?vnetId=4742&networkServiceProviderId=1] has no instance associated
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199) ~[guava-15.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3932) ~[guava-15.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936) ~[guava-15.0.jar:na]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806) ~[guava-15.0.jar:na]
at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4812) ~[guava-15.0.jar:na]

Tried the solutions on github but no luck that instance is set to status= purge

6

Conversation moved to github issue
https://github.com/rancher/rancher/issues/9316