Hi,
we have installed Cloud 2.0. Sometime(4 -5 per day) we have problem, that the chef,crowbar services will die.
Logs:
/var/log/chef/client.log
[Wed, 06 Nov 2013 08:23:47 +0000] INFO: Forked, in 3830. Privileges: 0 0
[Wed, 06 Nov 2013 08:23:48 +0000] INFO: Forking chef instance to converge…
[Wed, 06 Nov 2013 08:23:48 +0000] INFO: Fork successful. Waiting for new chef pid: 3832
[Wed, 06 Nov 2013 08:23:48 +0000] INFO: Forked instance now converging
[Wed, 06 Nov 2013 08:23:48 +0000] INFO: *** Chef 10.24.4 ***
[Wed, 06 Nov 2013 08:23:51 +0000] ERROR: Connection refused connecting to localhost:4000 for /nodes/bla, retry 1/5
[Wed, 06 Nov 2013 08:23:57 +0000] ERROR: Connection refused connecting to localhost:4000 for /nodes/bla, retry 2/5
[Wed, 06 Nov 2013 08:24:02 +0000] ERROR: Connection refused connecting to localhost:4000 for /nodes/bla, retry 3/5
[Wed, 06 Nov 2013 08:24:07 +0000] ERROR: Connection refused connecting to localhost:4000 for /nodes/bla, retry 4/5
[Wed, 06 Nov 2013 08:24:12 +0000] ERROR: Connection refused connecting to localhost:4000 for /nodes/bla, retry 5/5
[Wed, 06 Nov 2013 08:24:17 +0000] ERROR: RuntimeError: Forked convergence run failed
[Wed, 06 Nov 2013 08:24:17 +0000] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[Wed, 06 Nov 2013 08:24:17 +0000] ERROR: Sleeping for 900 seconds before trying again
[Wed, 06 Nov 2013 08:39:32 +0000] INFO: Forking chef instance to converge…
[Wed, 06 Nov 2013 08:39:32 +0000] INFO: Fork successful. Waiting for new chef pid: 5431
[Wed, 06 Nov 2013 08:39:32 +0000] INFO: Forked instance now converging
[Wed, 06 Nov 2013 08:39:32 +0000] INFO: *** Chef 10.24.4 ***
/var/chef/cache/chef-stacktrace.out:
Generated at Wed Nov 06 09:28:19 +0000 2013
Errno::ECONNRESET: Connection reset by peer
/usr/lib64/ruby/1.8/net/protocol.rb:135:in sysread' /usr/lib64/ruby/1.8/net/protocol.rb:135:in rbuf_fill’
/usr/lib64/ruby/1.8/timeout.rb:67:in timeout' /usr/lib64/ruby/1.8/timeout.rb:101:in timeout’
/usr/lib64/ruby/1.8/net/protocol.rb:134:in rbuf_fill' /usr/lib64/ruby/1.8/net/protocol.rb:116:in readuntil’
/usr/lib64/ruby/1.8/net/protocol.rb:126:in readline' /usr/lib64/ruby/1.8/net/http.rb:2028:in read_status_line’
/usr/lib64/ruby/1.8/net/http.rb:2017:in read_new' /usr/lib64/ruby/1.8/net/http.rb:1051:in request’
/usr/lib64/ruby/1.8/net/http.rb:1037:in request' /usr/lib64/ruby/1.8/net/http.rb:543:in start’
/usr/lib64/ruby/1.8/net/http.rb:1035:in request' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/rest/rest_request.rb:99:in call’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/rest/rest_request.rb:114:in hide_net_http_bug' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/rest/rest_request.rb:98:in call’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/rest.rb:258:in raw_http_request' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/rest.rb:370:in retriable_rest_request’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/rest.rb:256:in raw_http_request' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/rest.rb:250:in api_request’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/rest.rb:140:in get_rest' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/node.rb:604:in load’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/node.rb:588:in find_or_create' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/client.rb:294:in load_node’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/client.rb:409:in do_run' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/client.rb:176:in run’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/application.rb:140:in run_chef_client' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application/client.rb:274:in run_application’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/application/client.rb:267:in loop' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application/client.rb:267:in run_application’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/application.rb:72:in run' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/chef-client:26 /usr/bin/chef-client:19:in load’
/usr/bin/chef-client:19
Temporary solutions of this problem is restart chefs and crowbars services
In support case said me about this forum for solution. I know that the Cloud 2.0 is young product, but from the payed product this problems …
Many thanks for help
Karel Peran
