Chef and crowbar server problem

Hi,

we have installed Cloud 2.0. Sometime(4 -5 per day) we have problem, that the chef,crowbar services will die.

Logs:
/var/log/chef/client.log
[Wed, 06 Nov 2013 08:23:47 +0000] INFO: Forked, in 3830. Privileges: 0 0
[Wed, 06 Nov 2013 08:23:48 +0000] INFO: Forking chef instance to converge…
[Wed, 06 Nov 2013 08:23:48 +0000] INFO: Fork successful. Waiting for new chef pid: 3832
[Wed, 06 Nov 2013 08:23:48 +0000] INFO: Forked instance now converging
[Wed, 06 Nov 2013 08:23:48 +0000] INFO: *** Chef 10.24.4 ***
[Wed, 06 Nov 2013 08:23:51 +0000] ERROR: Connection refused connecting to localhost:4000 for /nodes/bla, retry 1/5
[Wed, 06 Nov 2013 08:23:57 +0000] ERROR: Connection refused connecting to localhost:4000 for /nodes/bla, retry 2/5
[Wed, 06 Nov 2013 08:24:02 +0000] ERROR: Connection refused connecting to localhost:4000 for /nodes/bla, retry 3/5
[Wed, 06 Nov 2013 08:24:07 +0000] ERROR: Connection refused connecting to localhost:4000 for /nodes/bla, retry 4/5
[Wed, 06 Nov 2013 08:24:12 +0000] ERROR: Connection refused connecting to localhost:4000 for /nodes/bla, retry 5/5
[Wed, 06 Nov 2013 08:24:17 +0000] ERROR: RuntimeError: Forked convergence run failed
[Wed, 06 Nov 2013 08:24:17 +0000] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[Wed, 06 Nov 2013 08:24:17 +0000] ERROR: Sleeping for 900 seconds before trying again
[Wed, 06 Nov 2013 08:39:32 +0000] INFO: Forking chef instance to converge…
[Wed, 06 Nov 2013 08:39:32 +0000] INFO: Fork successful. Waiting for new chef pid: 5431
[Wed, 06 Nov 2013 08:39:32 +0000] INFO: Forked instance now converging
[Wed, 06 Nov 2013 08:39:32 +0000] INFO: *** Chef 10.24.4 ***

/var/chef/cache/chef-stacktrace.out:
Generated at Wed Nov 06 09:28:19 +0000 2013
Errno::ECONNRESET: Connection reset by peer
/usr/lib64/ruby/1.8/net/protocol.rb:135:in sysread' /usr/lib64/ruby/1.8/net/protocol.rb:135:in rbuf_fill’
/usr/lib64/ruby/1.8/timeout.rb:67:in timeout' /usr/lib64/ruby/1.8/timeout.rb:101:in timeout’
/usr/lib64/ruby/1.8/net/protocol.rb:134:in rbuf_fill' /usr/lib64/ruby/1.8/net/protocol.rb:116:in readuntil’
/usr/lib64/ruby/1.8/net/protocol.rb:126:in readline' /usr/lib64/ruby/1.8/net/http.rb:2028:in read_status_line’
/usr/lib64/ruby/1.8/net/http.rb:2017:in read_new' /usr/lib64/ruby/1.8/net/http.rb:1051:in request’
/usr/lib64/ruby/1.8/net/http.rb:1037:in request' /usr/lib64/ruby/1.8/net/http.rb:543:in start’
/usr/lib64/ruby/1.8/net/http.rb:1035:in request' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/rest/rest_request.rb:99:in call’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/rest/rest_request.rb:114:in hide_net_http_bug' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/rest/rest_request.rb:98:in call’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/rest.rb:258:in raw_http_request' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/rest.rb:370:in retriable_rest_request’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/rest.rb:256:in raw_http_request' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/rest.rb:250:in api_request’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/rest.rb:140:in get_rest' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/node.rb:604:in load’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/node.rb:588:in find_or_create' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/client.rb:294:in load_node’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/client.rb:409:in do_run' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/client.rb:176:in run’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/application.rb:140:in run_chef_client' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application/client.rb:274:in run_application’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/application/client.rb:267:in loop' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application/client.rb:267:in run_application’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/application.rb:72:in run' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/chef-client:26 /usr/bin/chef-client:19:in load’
/usr/bin/chef-client:19

Temporary solutions of this problem is restart chefs and crowbars services

In support case said me about this forum for solution. I know that the Cloud 2.0 is young product, but from the payed product this problems …

Many thanks for help

Karel Peran

The same problem, but with different stacktrace

/var/chef/cache/chef-stacktrace.out
Generated at Wed Nov 06 10:18:10 +0000 2013
RuntimeError: Forked convergence run failed
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/client.rb:172:in run' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application.rb:140:in run_chef_client’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/application/client.rb:274:in run_application' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application/client.rb:267:in loop’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/…/lib/chef/application/client.rb:267:in run_application' /usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/../lib/chef/application.rb:72:in run’
/usr/lib64/ruby/gems/1.8/gems/chef-10.24.4/bin/chef-client:26
/usr/bin/chef-client:19:in `load’
/usr/bin/chef-client:19

Hi Karel,

if that was a service request you opened with SuSE, would you mind sending me the SR number via personal message?

Concerning the error, I’ll have to investigate further and so invite anyone to jump at this thread :wink:

Regards,
Jens

Hi Karel,

I’ll get in touch with you via PM, sending you a support contact within SUSE/Novell.

With regards,
Jens

Hi All,

Are you using SMT? and if so did you do an update on SUSE Cloud?

Regards
Johann

On Thu, 03 Apr 2014 09:14:01 +0000, jels wrote:
[color=blue]

Hi All,

Are you using SMT? and if so did you do an update on SUSE Cloud?

Regards Johann[/color]

Dear Johann,

The ‘Chef and crowbar server problem’ got resolved.

Best regards
Hans