Upgrade issue from 0.63.0 to 1.0.0

Rancher 1.x
1

Hi,

We get a few errors in the server log after upgrade:

07:48:22.217 [main] INFO  ConsoleStatus - [DONE ] [260146ms] Startup Succeeded, Listening on port 8081
time="2016-03-29T07:48:22Z" level=info msg="Starting websocket proxy. Listening on [:8080], Proxying to cattle API at [localhost:8081], Monitoring parent pid [8]." 
time="2016-03-29T07:48:23Z" level=info msg="Downloading certificate from http://localhost:8081/v1/credentials/1c2/certificate" 
time="2016-03-29T07:48:24Z" level=info msg="Starting Rancher Catalog service" 
time="2016-03-29T07:48:24Z" level=info msg="Using catalog library=http://git.xxx/yyy/rancher-catalog.git" 
time="2016-03-29T07:48:24Z" level=info msg="Using catalog community=http://git.xxx/yyy/rancher-community-catalog.git" 
Cloning into './DATA/community'...
time="2016-03-29T07:48:24Z" level=info msg="Cloning the catalog from git URL http://git.xxx/yyy/rancher-community-catalog.git" 
time="2016-03-29T07:48:24Z" level=info msg="Starting rancher-compose-executor" version=v0.7.3 
time="2016-03-29T07:48:24Z" level=info msg="Setting log level" logLevel=info 
time="2016-03-29T07:48:24Z" level=info msg="Starting go-machine-service..." gitcommit=dc97268 
time="2016-03-29T07:48:24Z" level=info msg="Updating docker-machine-drivers from cattle." 
time="2016-03-29T07:48:24Z" level=info msg="Initializing event router" workerCount=10 
time="2016-03-29T07:48:24Z" level=error msg="Driver: ubiquity is error ignoring driver download." 
time="2016-03-29T07:48:24Z" level=error msg="Driver: packet is error ignoring driver download." 
time="2016-03-29T07:48:24Z" level=info msg="packet not to be used removing any schemas it has." 
time="2016-03-29T07:48:24Z" level=info msg="ubiquity not to be used removing any schemas it has." 
time="2016-03-29T07:48:24Z" level=info msg="Connection established" 
time="2016-03-29T07:48:24Z" level=info msg="Cloning the catalog from git URL http://git.xxx/yyy/rancher-catalog.git" 
Cloning into './DATA/library'...
time="2016-03-29T07:48:28Z" level=error msg="Encountered multiple errors while updating drivers." 
time="2016-03-29T07:48:28Z" level=error msg="Error from driver: Failed when uploading genericConfig schema to cattle: Bad response statusCode [422]. Status [422 status code 422]. Body: [fieldName=SchemaExistsForGivenRoleAndOrId, code=NotUnique] from [http://localhost:8080/v1/dynamicschemas]" 
time="2016-03-29T07:48:28Z" level=error msg="Error from driver: Failed when uploading rackspaceConfig schema to cattle: Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=SchemaExistsForGivenRoleAndOrId] from [http://localhost:8080/v1/dynamicschemas]" 
time="2016-03-29T07:48:28Z" level=error msg="Error from driver: Failed when uploading hypervConfig schema to cattle: Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=SchemaExistsForGivenRoleAndOrId] from [http://localhost:8080/v1/dynamicschemas]" 
time="2016-03-29T07:48:28Z" level=error msg="Error from driver: Failed when uploading azureConfig schema to cattle: Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=SchemaExistsForGivenRoleAndOrId] from [http://localhost:8080/v1/dynamicschemas]" 
time="2016-03-29T07:48:28Z" level=error msg="Error from driver: Failed when uploading virtualboxConfig schema to cattle: Bad response statusCode [422]. Status [422 status code 422]. Body: [fieldName=SchemaExistsForGivenRoleAndOrId, code=NotUnique] from [http://localhost:8080/v1/dynamicschemas]" 
time="2016-03-29T07:48:28Z" level=error msg="Error from driver: Failed when uploading googleConfig schema to cattle: Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=SchemaExistsForGivenRoleAndOrId] from [http://localhost:8080/v1/dynamicschemas]" 
time="2016-03-29T07:48:28Z" level=error msg="Error from driver: Failed when uploading vmwarevsphereConfig schema to cattle: Bad response statusCode [422]. Status [422 status code 422]. Body: [fieldName=SchemaExistsForGivenRoleAndOrId, code=NotUnique] from [http://localhost:8080/v1/dynamicschemas]" 
time="2016-03-29T07:48:28Z" level=error msg="Error from driver: Failed when uploading vmwarevcloudairConfig schema to cattle: Bad response statusCode [422]. Status [422 status code 422]. Body: [fieldName=SchemaExistsForGivenRoleAndOrId, code=NotUnique] from [http://localhost:8080/v1/dynamicschemas]" 
time="2016-03-29T07:48:28Z" level=error msg="Error from driver: Failed when uploading exoscaleConfig schema to cattle: Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=SchemaExistsForGivenRoleAndOrId] from [http://localhost:8080/v1/dynamicschemas]" 
time="2016-03-29T07:48:28Z" level=error msg="Error from driver: Failed when uploading vmwarefusionConfig schema to cattle: Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=SchemaExistsForGivenRoleAndOrId] from [http://localhost:8080/v1/dynamicschemas]" 
time="2016-03-29T07:48:28Z" level=error msg="Error from driver: Failed when uploading softlayerConfig schema to cattle: Bad response statusCode [422]. Status [422 status code 422]. Body: [fieldName=SchemaExistsForGivenRoleAndOrId, code=NotUnique] from [http://localhost:8080/v1/dynamicschemas]" 
time="2016-03-29T07:48:28Z" level=error msg="Error from driver: Failed when uploading digitaloceanConfig schema to cattle: Bad response statusCode [422]. Status [422 status code 422]. Body: [fieldName=SchemaExistsForGivenRoleAndOrId, code=NotUnique] from [http://localhost:8080/v1/dynamicschemas]" 
time="2016-03-29T07:48:28Z" level=error msg="Error from driver: Failed when uploading amazonec2Config schema to cattle: Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=SchemaExistsForGivenRoleAndOrId] from [http://localhost:8080/v1/dynamicschemas]" 
time="2016-03-29T07:48:28Z" level=error msg="Error from driver: Failed when uploading openstackConfig schema to cattle: Bad response statusCode [422]. Status [422 status code 422]. Body: [code=NotUnique, fieldName=SchemaExistsForGivenRoleAndOrId] from [http://localhost:8080/v1/dynamicschemas]" 
time="2016-03-29T07:48:28Z" level=fatal msg="Encourted an error while updating drivers."

Still looking into this, just wanted to post it asap, in case it is of any significance…

EDIT: It’s still spamming these errors in the logs…

2

Took a dive and cleared two tables (while server being down): dynamic_schema and dynamic_schema_role, then started the server again. And this time, it came up without all the error messages from the driver, complaining about failed uploads.
The only related messages I got, and only once during boot, was:

time="2016-03-29T10:07:37Z" level=info msg="Updating docker-machine-drivers from cattle." 
time="2016-03-29T10:07:38Z" level=info msg="Initializing event router" workerCount=10 
time="2016-03-29T10:07:38Z" level=error msg="Driver: ubiquity is error ignoring driver download." 
time="2016-03-29T10:07:38Z" level=error msg="Driver: packet is error ignoring driver download." 
time="2016-03-29T10:07:38Z" level=info msg="packet not to be used removing any schemas it has." 
time="2016-03-29T10:07:38Z" level=info msg="ubiquity not to be used removing any schemas it has."
3

Thanks @kaos. We’ll also look into why this is possibly happening.

4

@kaos Can you provide more details on your setup?
What version of Docker?
What kind of authentication?

Had you done any upgrades/downgrades previously to any of the v1.0.0-rcX releases?

5

Hi @denise,

I sent more details and logs in a mail to @willchan. I missed to include the answers to auth and upgrade history.

We use LDAP authentication, and was going from 0.63.0 to 1.0.0 when I got those messages in the log. I did make an attempt after that going back to 0.63.0, but seeing that it didn’t resolve this particular issue, I went back on 1.0.0, wiped the two dynmic_schema and dynmic_schema_roles tables and it went away. But we’re still having other issues, as per my email to Will.