General or per host "newsfeed" & "ghost containers"

Hi,

I have a suggestion and also a weird case to reprt… The suggestion is to have a place to view a container newsfeed, or an entire host’s “newsfeed” or environment “news feed”… basically it is similar to the logs, but showing information from Rancher and the container launch process… I see the container “step” whenever I launch a container displayed under the container name in the hosts view, but its really small field showing only the first part of whatever is happening… I’d like to be able to see the entire process, and also to be able to look back a bit…

One situation which I see happening is that sometimes containers are launched over and over, and at one point become “orphans” or “ghosts”…

Is there any way to remove containers in mass?

If I look inside one of these containers, its apparently a failed build command from the docker image… I’m unsure why they got there (esp since I’m always using rancher-compose to send a complete stack), but sometimes I do notice a container being created/recreated during a build process… Usually the “old one” gets purged out, but something happened…

Further on my “general log” idea it would be interesting (IMO) to see a combination of what rancher receives as commands (this is already in the audit log), but also what happens after… a sort of semi-verbose log aggregation of what rancher is doing… sometimes I think a service failed to launch, and then a couple of minutes later it starts up the containers, or missing containers… I’m always a bit stumped: was rancher doing something? was a timeout triggered? was it a healthcheck that made it “heal” the broken service? Some sort of visibility would be nice I think? If others are accomplishing this in some other way please share :slight_smile:

Still getting “ghost containers”… I think they may be related to the build process, or to the S3 build process in particular…

I started from scratch and already have since starting tests this morning on the latest version abt 15 “ghost” containers… some containers are added and then trashed in the build process, others are just being added and “sticking”…

If I look into the logs, here are some examples, which seem to validate my build theory…


What version of rancher-compose are you using?

We fixed this issue where we weren’t removing containers during the build.

But your issue might cause ghost containers to be created:

Hi @denise I’m not sure what version I was on last week, but it was downloaded from the UI… right now I’m on:

(I did update rancher-compose cli when I created the rancher/server and rancher agent machines…)

I’m not getting the reconnecting state as long as I use another OS for host. Last time I checked using rancheros it would always knock it out… So I switched to core-os for my tests…

I’ve just noticed the release notes state incompatibility with coreos… I missed that… I’ll hold off in testing until another release is out then… I’m seeing connectivity issues between hosts which I guess I’ll attribute to this…

@denise just an FYI, I’m still getting “ghost” containers on 0.56.1. This is now again on CoreOS and the hosts arent in reconnecting or anything…

I’m not sure how you didn’t hit this issue. :slight_smile: Anytime I try to run a build, I end up getting my hosts into reconnecting and because of the reconnecting stage, I’ll get some random containers from it.

I get hosts going to reconnecting whenever I try using rancheros, but with coreos it was ok… I hope I dont jynx myself :stuck_out_tongue: