Hello There,
I’ve been working with Rancher since a long and planning to have Rancher as our backend container orchestration on Amazon.
We have automated our infrastructure on aws with cloudformation have images for Rancher and worker. So the Rancher image will have a rancher-server running. When we scale up instances the worker will run the custom docker command to get added to Rancher with the help of UserData. Everything looks good but the worker doesn’t add to Rancher. Logs looks fine nothing unusual. I need some help in solving this isssue and this is a blocker at this moment for us to proceed with it. Below logs might give you some Idea.
On Worker Machine
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d6fe0fba76d3 rancher/agent:v0.8.2 "/run.sh run" 13 seconds ago Up 12 seconds rancher-agent
root@ip-10-0-3-14:~# docker logs -f d6fe0fba76d3
INFO: Starting agent for B5C9DF41BB9AB4C5A940
INFO: Access Key: B5C9DF41BB9AB4C5A940
INFO: Config URL: http://rancher-master.synduit4.com/v1
INFO: Storage URL: http://rancher-master.synduit4.com/v1
INFO: API URL: http://rancher-master.synduit4.com/v1
INFO: IP: 54.201.33.84
INFO: Port:
INFO: Required Image: rancher/agent:v0.8.2
INFO: Current Image: rancher/agent:v0.8.2
INFO: Using image rancher/agent:v0.8.2
INFO: Downloading agent http://rancher-master.synduit4.com/v1/configcontent/configscripts
INFO: Updating configscripts
INFO: Downloading http://rancher-master.synduit4.com/v1//configcontent//configscripts current=
INFO: Running /var/lib/cattle/download/configscripts/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/apply.sh
INFO: Sending configscripts applied 1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319
INFO: Updating pyagent
INFO: Downloading http://rancher-master.synduit4.com/v1//configcontent//pyagent current=
INFO: Running /var/lib/cattle/download/pyagent/pyagent-1-9e3dcab12585149a65d818082279708965cd826df1937a55b24f48428f8f0c06069547103e24f5f4562a2db5d4805d60/apply.sh
INFO: Updating host-api
INFO: Downloading http://rancher-master.synduit4.com/v1//configcontent//host-api current=1-1dcc85d77415faddc4d6d204d2bd622d835de4b5ee4b472651419b58189c474a
INFO: Already up to date
INFO: Sending host-api applied 1-1dcc85d77415faddc4d6d204d2bd622d835de4b5ee4b472651419b58189c474a
INFO: Updating agent-binaries
INFO: Downloading http://rancher-master.synduit4.com/v1//configcontent//agent-binaries current=1-4200f9b686342da8e941359211b1d8c5f98877ae293d0ead231d43507f1c9eae
INFO: Already up to date
INFO: Sending agent-binaries applied 1-4200f9b686342da8e941359211b1d8c5f98877ae293d0ead231d43507f1c9eae
INFO: Updating python-agent
INFO: Downloading http://rancher-master.synduit4.com/v1//configcontent//python-agent current=1-5fa9b53964d8b1335257916f7b4687aec2e17c9516e578f5406f29160eec1cd1
INFO: Already up to date
INFO: Sending python-agent applied 1-5fa9b53964d8b1335257916f7b4687aec2e17c9516e578f5406f29160eec1cd1
INFO: Updating cadvisor
INFO: Downloading http://rancher-master.synduit4.com/v1//configcontent//cadvisor current=1-4d2b4356ab14bc1fa7ec7cd83f35b7e975a16cea01634683b577c9498c98de25
INFO: Already up to date
INFO: Sending cadvisor applied 1-4d2b4356ab14bc1fa7ec7cd83f35b7e975a16cea01634683b577c9498c98de25
INFO: Updating volmgr
INFO: Downloading http://rancher-master.synduit4.com/v1//configcontent//volmgr current=1-c813690c24ae9a12f0792378c78a48f447b243517233d7969b99baefbea5af15
INFO: Already up to date
INFO: Sending volmgr applied 1-c813690c24ae9a12f0792378c78a48f447b243517233d7969b99baefbea5af15
INFO: Sending pyagent applied 1-9e3dcab12585149a65d818082279708965cd826df1937a55b24f48428f8f0c06069547103e24f5f4562a2db5d4805d60
INFO: Starting agent /var/lib/cattle/pyagent/apply.sh
INFO: Executing /var/lib/cattle/pyagent/main.py
INFO: Updating host-config
INFO: Downloading http://rancher-master.synduit4.com/v1//configcontent//host-config current=host-config-1-640500f09b9fd139fb5a9533f59beb3d214ba5bba5add601eebf4fdf4db858c5
INFO: Already up to date
INFO: Sending host-config applied 1-640500f09b9fd139fb5a9533f59beb3d214ba5bba5add601eebf4fdf4db858c5
time="2015-12-30T07:41:33Z" level="info" msg="Starting event router."
time="2015-12-30T07:41:33Z" level="info" msg="Watching state directory: /var/lib/rancher/state/containers"
time="2015-12-30T07:41:33Z" level="info" msg="Processing event: &docker.APIEvents{Status:\"start\", ID:\"d6fe0fba76d3082be604b47430950a519067dc3444fc490db91a0d36952d6f2e\", From:\"-simulated-\", Time:0}"
time="2015-12-30T07:41:33Z" level="info" msg="Connecting to proxy." url="ws://rancher-master.synduit4.com/v1/connectbackend?token=eyJhbGciOiJSUzI1NiJ9.eyJleHAiOjE0NTE0NjE1OTIsInN1YiI6ImNhdHRsZSIsImlzcyI6Imh0dHA6XC9cL2NhdHRsZS5pbyIsInJlcG9ydGVkVXVpZCI6ImZjMzFhOTUwLTVhNGUtNDFlYy05YjFkLTBkNWY2MzFlNTQ3MSIsImlhdCI6MTQ1MTQ2MTI5Miwia2lkIjoiZGVmYXVsdCJ9.DXge9DtbAtYQUfRebjiv7CGYU4pnBc4RmRiJg1Qm-fsb3Xxj9604YJBloGHiJK3q1760qesvF5Vs0I4BynFRKLrip0fhmMjtmwG9uPsadHPO7WdPTAIBkFtl_kXvq-esGyCXJ4D80fxjGRMeZsx6jjrwSNb8cQ_GkHCDoAPDFrVt9M5sUNDiseeJrMYf-Ju2hhQ-VRBirrxzVXxLVORKqcjD_yP1gTXZIjK2Ls7mqta1hkqHUg45MQd_jDm8kuBBhZwa9oW3F4bq-pUy3WR0QbDrKmqCtB8L0swrc4TaY7jLxeKtAyhg_r1BvlCS84-pL3uK-66XOChc_c98uVCF6Q"
I1230 07:41:33.231007 31464 storagedriver.go:132] No backend storage selected
I1230 07:41:33.231048 31464 storagedriver.go:134] Caching stats in memory for 2m0s
I1230 07:41:33.231111 31464 manager.go:128] cAdvisor running in container: "/docker/d6fe0fba76d3082be604b47430950a519067dc3444fc490db91a0d36952d6f2e"
I1230 07:41:33.327571 31464 fs.go:108] Filesystem partitions: map[/dev/disk/by-uuid/c70a26ec-1dda-455c-acfd-792015b2bb6f:{mountpoint:/ major:202 minor:1 fsType: blockSize:0}]
I1230 07:41:33.333899 31464 manager.go:163] Machine: {NumCores:2 CpuFrequency:2400040 MemoryCapacity:8371847168 MachineID:e953ce35510a043907b5de705512a156 SystemUUID:EC24026F-8E07-DA7A-2468-012445547E3D BootID:5124364b-e1a0-4a4c-8bc3-945807dd59ef Filesystems:[{Device:/dev/disk/by-uuid/c70a26ec-1dda-455c-acfd-792015b2bb6f Capacity:42127835136}] DiskMap:map[202:0:{Name:xvda Major:202 Minor:0 Size:42949672960 Scheduler:deadline}] NetworkDevices:[{Name:eth0 MacAddress:06:fb:19:1c:35:b7 Speed:10000 Mtu:9001}] Topology:[{Id:0 Memory:8371847168 Cores:[{Id:0 Threads:[0 1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:31457280 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown}
I1230 07:41:33.346736 31464 manager.go:169] Version: {KernelVersion:3.13.0-74-generic ContainerOsVersion:Ubuntu 14.04.2 LTS DockerVersion:1.9.1 CadvisorVersion: CadvisorRevision:}
I1230 07:41:33.442298 31464 factory.go:236] Registering Docker factory
I1230 07:41:33.442792 31464 factory.go:93] Registering Raw factory
I1230 07:41:33.559463 31464 manager.go:1006] Started watching for new ooms in manager
I1230 07:41:33.559510 31464 oomparser.go:199] OOM parser using kernel log file: "/var/log/kern.log"
I1230 07:41:33.574194 31464 manager.go:250] Starting recovery of all containers
I1230 07:41:33.586418 31464 manager.go:255] Recovery completed
I1230 07:41:33.590533 31464 cadvisor.go:96] Starting cAdvisor version: - on port 9344
On Rancher
time="2015-12-30T07:41:32Z" level=info msg="Registering backend for host [fc31a950-5a4e-41ec-9b1d-0d5f631e5471]" time="2015-12-30T07:44:30Z" level=info msg="Pulling the catalog library from the repo to sync any new changes to ./DATA/library" time="2015-12-30T07:49:30Z" level=info msg="Pulling the catalog library from the repo to sync any new changes to ./DATA/library"
Thank You,
Swaroop Kundeti,
Axelerant Technologies.