Rancher API not returning the port number in links

This is a very curious one which has just started happening for no discernable reason. The API is not including the port number in the links. See here ```
Calling rancher on http://rancher-test.pibenchmark.com:8080/v1/projects
[type] => collection
[data] => table: 0xbf0e50 {
[1] => table: 0xbf1cd0 {
=> function: 0xbd9d80
[description] => function: 0xbd9d80
[transitioningMessage] => function: 0xbd9d80
[transitioning] => no
[virtualMachine] => false
[actions] => table: 0xbf9f30 {
}
[mesos] => false
[type] => project
[kubernetes] => false
[createdTS] => 1464691747000
[id] => 1a5
[uuid] => adminProject
[members] => function: 0xbd9d80
[links] => table: 0xbf9f30 {
[externalDnsEvents] => http://rancher-test.pibenchmark.com/v1/projects/1a5/externaldnsevents
[ipAddresses] => http://rancher-test.pibenchmark.com/v1/projects/1a5/ipaddresses
[amazonec2Configs] => http://rancher-test.pibenchmark.com/v1/projects/1a5/amazonec2configs
[auditLogs] => http://rancher-test.pibenchmark.com/v1/projects/1a5/auditlogs
[registryCredentials] => http://rancher-test.pibenchmark.com/v1/projects/1a5/registrycredentials
[registries] => http://rancher-test.pibenchmark.com/v1/projects/1a5/registries
[machineDrivers] => http://rancher-test.pibenchmark.com/v1/projects/1a5/machinedrivers
[storagePools] => http://rancher-test.pibenchmark.com/v1/projects/1a5/storagepools
[projects] => http://rancher-test.pibenchmark.com/v1/projects/1a5/projects


Oddly the port is included when using the API via a browser. This was working completely fine until a few minutes ago. I'm completely baffled

Is this still happening in your setup? I don’t typically use the API, but it’d be nice it there was something triggering it.

Yep, it’s still doing it. I’ll try and unpick my library to show you what exactly I’m doing.

This is a demonstration of what I’m doing. It’s in LUA just to be weird, mainly as it plugs in to Nginx.

function print_r ( t )
    local print_r_cache={}
    local function sub_print_r(t,indent)
        if (print_r_cache[tostring(t)]) then
            print(indent.."*"..tostring(t))
        else
            print_r_cache[tostring(t)]=true
            if (type(t)=="table") then
                for pos,val in pairs(t) do
                    if (type(val)=="table") then
                        print(indent.."["..pos.."] => "..tostring(t).." {")
                        sub_print_r(val,indent..string.rep(" ",string.len(pos)+8))
                        print(indent..string.rep(" ",string.len(pos)+6).."}")
                    else
                        print(indent.."["..pos.."] => "..tostring(val))
                    end
                end
            else
                print(indent..tostring(t))
            end
        end
    end
    sub_print_r(t,"  ")
end

function getApiKeys()

        return { test = { rancher_url = 'http://rancher-test.pibenchmark.com:8080',
                          rancher_access_key = 'xxxxxxxxxxxxxxxxx',
                          rancher_secret_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' }
               }

end

-- make a rancher url
function rancherUrl(args,uri)

        api_keys = getApiKeys()

        if api_keys[args['rancher_env']]['rancher_url'] == Nil then
                print("No API keys for environment "..args['rancher_env'])
        end

        url = api_keys[args['rancher_env']]['rancher_url']..uri

        print("Rancher URL = "..url)

        return url

end

-- general purpose rancher call
function rancherCall(args, url)

        api_keys = getApiKeys()

        local http  = require "socket.http"
        local mime  = require "mime"
        local ltn12 = require "ltn12"
        local json  = require "json"

        response = {}
        auth = mime.b64(api_keys[args['rancher_env']]['rancher_access_key']..":"
                      ..api_keys[args['rancher_env']]['rancher_secret_key'])
        sink = ltn12.sink.table(response)
        void, status = http.request{ url = url, sink = sink, headers = { Authorization = ("Basic "..auth) } }

        data = json.decode(table.concat(response))
        return status, data

end

args = {rancher_env = 'test'}
status, result = rancherCall(args, rancherUrl(args,'/v1/projects'))

print("Status is "..status)
print_r(result)

And the results come back without the port number

Rancher URL = http://rancher-test.pibenchmark.com:8080/v1/projects
Status is 200
  [type] => collection
  [data] => table: 0x1d05160 {
              [1] => table: 0x1d07f90 {
                        => function: 0x1ce9e50
                       [description] => function: 0x1ce9e50
                       [transitioningMessage] => function: 0x1ce9e50
                       [transitioning] => no
                       [virtualMachine] => false
                       [actions] => table: 0x1d08240 {
                                    }
                       [mesos] => false
                       [type] => project
                       [kubernetes] => false
                       [createdTS] => 1464691747000
                       [id] => 1a5
                       [uuid] => adminProject
                       [members] => function: 0x1ce9e50
                       [links] => table: 0x1d08240 {
                                    [externalDnsEvents] => http://rancher-test.pibenchmark.com/v1/projects/1a5/externaldnsevents
                                    [ipAddresses] => http://rancher-test.pibenchmark.com/v1/projects/1a5/ipaddresses
                                    [amazonec2Configs] => http://rancher-test.pibenchmark.com/v1/projects/1a5/amazonec2configs
                                    [auditLogs] => http://rancher-test.pibenchmark.com/v1/projects/1a5/auditlogs
                                    [registryCredentials] => http://rancher-test.pibenchmark.com/v1/projects/1a5/registrycredentials
                                    [registries] => http://rancher-test.pibenchmark.com/v1/projects/1a5/registries
                                    [machineDrivers] => http://rancher-test.pibenchmark.com/v1/projects/1a5/machinedrivers
                                    [storagePools] => http://rancher-test.pibenchmark.com/v1/projects/1a5/storagepools
                                    [projects] => http://rancher-test.pibenchmark.com/v1/projects/1a5/projects
                                    [serviceExposeMaps] => http://rancher-test.pibenchmark.com/v1/projects/1a5/serviceexposemaps
                                    [dnsServices] => http://rancher-test.pibenchmark.com/v1/projects/1a5/dnsservices
                                    [containerExecs] => http://rancher-test.pibenchmark.com/v1/projects/1a5/containerexecs
                                    [composeProjects] => http://rancher-test.pibenchmark.com/v1/projects/1a5/composeprojects
                                    [composeServices] => http://rancher-test.pibenchmark.com/v1/projects/1a5/composeservices
                                    [credentials] => http://rancher-test.pibenchmark.com/v1/projects/1a5/credentials
                                    [ports] => http://rancher-test.pibenchmark.com/v1/projects/1a5/ports
                                    [machines] => http://rancher-test.pibenchmark.com/v1/projects/1a5/machines
                                    [serviceConsumeMaps] => http://rancher-test.pibenchmark.com/v1/projects/1a5/serviceconsumemaps
                                    [azureConfigs] => http://rancher-test.pibenchmark.com/v1/projects/1a5/azureconfigs
                                    [labels] => http://rancher-test.pibenchmark.com/v1/projects/1a5/labels
                                    [digitaloceanConfigs] => http://rancher-test.pibenchmark.com/v1/projects/1a5/digitaloceanconfigs
                                    [images] => http://rancher-test.pibenchmark.com/v1/projects/1a5/images
                                    [passwords] => http://rancher-test.pibenchmark.com/v1/projects/1a5/passwords
                                    [instances] => http://rancher-test.pibenchmark.com/v1/projects/1a5/instances
                                    [serviceEvents] => http://rancher-test.pibenchmark.com/v1/projects/1a5/serviceevents
                                    [externalVolumeEvents] => http://rancher-test.pibenchmark.com/v1/projects/1a5/externalvolumeevents
                                    [packetConfigs] => http://rancher-test.pibenchmark.com/v1/projects/1a5/packetconfigs
                                    [apiKeys] => http://rancher-test.pibenchmark.com/v1/projects/1a5/apikeys
                                    [typeDocumentations] => http://rancher-test.pibenchmark.com/v1/projects/1a5/typedocumentations
                                    [addOutputsInputs] => http://rancher-test.pibenchmark.com/v1/projects/1a5/addoutputsinputs
                                    [certificates] => http://rancher-test.pibenchmark.com/v1/projects/1a5/certificates
                                    [healthcheckInstanceHostMaps] => http://rancher-test.pibenchmark.com/v1/projects/1a5/healthcheckinstancehostmaps
                                    [environmentUpgrades] => http://rancher-test.pibenchmark.com/v1/projects/1a5/environmentupgrades
                                    [externalServiceEvents] => http://rancher-test.pibenchmark.com/v1/projects/1a5/externalserviceevents
                                    [virtualMachines] => http://rancher-test.pibenchmark.com/v1/projects/1a5/virtualmachines
                                    [accounts] => http://rancher-test.pibenchmark.com/v1/projects/1a5/accounts
                                    [externalEvents] => http://rancher-test.pibenchmark.com/v1/projects/1a5/externalevents
                                    [containerEvents] => http://rancher-test.pibenchmark.com/v1/projects/1a5/containerevents
                                    [instanceLinks] => http://rancher-test.pibenchmark.com/v1/projects/1a5/instancelinks
                                    [hostStats] => http://rancher-test.pibenchmark.com/v1/projects/1a5/projects/1a5/hoststats
                                    [externalServices] => http://rancher-test.pibenchmark.com/v1/projects/1a5/externalservices
                                    [physicalHosts] => http://rancher-test.pibenchmark.com/v1/projects/1a5/physicalhosts
                                    [hosts] => http://rancher-test.pibenchmark.com/v1/projects/1a5/hosts
                                    [dockerBuilds] => http://rancher-test.pibenchmark.com/v1/projects/1a5/dockerbuilds
                                    [kubernetesServices] => http://rancher-test.pibenchmark.com/v1/projects/1a5/kubernetesservices
                                    [statsAccesses] => http://rancher-test.pibenchmark.com/v1/projects/1a5/statsaccesses
                                    [settings] => http://rancher-test.pibenchmark.com/v1/projects/1a5/settings
                                    [containers] => http://rancher-test.pibenchmark.com/v1/projects/1a5/containers
                                    [hostAccesses] => http://rancher-test.pibenchmark.com/v1/projects/1a5/hostaccesses
                                    [environments] => http://rancher-test.pibenchmark.com/v1/projects/1a5/environments
                                    [loadBalancerServices] => http://rancher-test.pibenchmark.com/v1/projects/1a5/loadbalancerservices
                                    [schemas] => http://rancher-test.pibenchmark.com/v1/projects/1a5/schemas
                                    [projectMembers] => http://rancher-test.pibenchmark.com/v1/projects/1a5/projectmembers
                                    [register] => http://rancher-test.pibenchmark.com/v1/projects/1a5/register
                                    [serviceProxies] => http://rancher-test.pibenchmark.com/v1/projects/1a5/serviceproxies
                                    [snapshots] => http://rancher-test.pibenchmark.com/v1/projects/1a5/snapshots
                                    [loadBalancerConfigs] => http://rancher-test.pibenchmark.com/v1/projects/1a5/loadbalancerconfigs
                                    [registrationTokens] => http://rancher-test.pibenchmark.com/v1/projects/1a5/registrationtokens
                                    [pullTasks] => http://rancher-test.pibenchmark.com/v1/projects/1a5/pulltasks
                                    [networks] => http://rancher-test.pibenchmark.com/v1/projects/1a5/networks
                                    [volumes] => http://rancher-test.pibenchmark.com/v1/projects/1a5/volumes
                                    [mounts] => http://rancher-test.pibenchmark.com/v1/projects/1a5/mounts
                                    [identities] => http://rancher-test.pibenchmark.com/v1/projects/1a5/identities
                                    [subscribe] => http://rancher-test.pibenchmark.com/v1/projects/1a5/subscribe
                                    [services] => http://rancher-test.pibenchmark.com/v1/projects/1a5/services
                                    [externalStoragePoolEvents] => http://rancher-test.pibenchmark.com/v1/projects/1a5/externalstoragepoolevents
                                    [externalHostEvents] => http://rancher-test.pibenchmark.com/v1/projects/1a5/externalhostevents
                                    [self] => http://rancher-test.pibenchmark.com/v1/projects/1a5
                                  }
                       [state] => active
                       [transitioningProgress] => function: 0x1ce9e50
                       [name] => Test
                       [servicesPortRange] => table: 0x1d08240 {
                                                [startPort] => 49153
                                                [endPort] => 65535
                                              }
                       [swarm] => false
                       [kind] => project
                       [created] => 2016-05-31T10:49:07Z
                       [publicDns] => false
                     }
            }
  [links] => table: 0x1d05160 {
               [self] => http://rancher-test.pibenchmark.com/v1/projects
             }
  [createDefaults] => table: 0x1d05160 {
                      }
  [resourceType] => project
  [pagination] => function: 0x1ce9e50
  [sortLinks] => table: 0x1d05160 {
                 }
  [createTypes] => table: 0x1d05160 {
                   }
  [sort] => function: 0x1ce9e50
  [actions] => table: 0x1d05160 {
               }
  [filters] => table: 0x1d05160 {
               }

Do you have a proxy or load balancer between the client and the rancher/server container? The API generates absolute URLs using the host header + X-Forwarded-{Host,Port,Proto}, so the only obvious way for this to happen is if the headers being received are wrong.

I suppose it’s also possible that your Lua HTTP client correctly makes a TCP connection to 8080 but incorrectly sends the Host header as “cher-test.pibenchmark.com” instead of “cher-test.pibenchmark.com:8080”.

Either way a tcpdump from the server container or the host it’s on should be helpful.

Sorry about the long delay, was on leave and had little access to servers.

Indeed the Lua socket library doesn’t send the port number in the Host header by default. I can work around it by overriding the headers but it’s annoying.

Here was the packet before my workaround:

15:32:31.927303 IP (tos 0x0, ttl 64, id 21145, offset 0, flags [DF], proto TCP (6), length 265)
    10.20.13.11.57769 > 10.20.13.202.8080: Flags [P.], cksum 0xd4f7 (correct), seq 826204422:826204635, ack 620334431, win 115, options [nop,nop,TS val 3614749308 ecr 2951538879], length 213
        0x0000:  4500 0109 5299 4000 4006 b859 0a14 0d0b  E...R.@.@..Y....
        0x0010:  0a14 0dca e1a9 1f90 313e e106 24f9 8d5f  ........1>..$.._
        0x0020:  8018 0073 d4f7 0000 0101 080a d774 b27c  ...s.........t.|
        0x0030:  afec e8bf 436f 6e6e 6563 7469 6f6e 3a20  ....Connection:.
        0x0040:  636c 6f73 652c 2054 450d 0a55 7365 722d  close,.TE..User-
        0x0050:  4167 656e 743a 204c 7561 536f 636b 6574  Agent:.LuaSocket
        0x0060:  2033 2e30 2d72 6331 0d0a 5445 3a20 7472  .3.0-rc1..TE:.tr
        0x0070:  6169 6c65 7273 0d0a 4175 7468 6f72 697a  ailers..Authoriz
        0x0080:  6174 696f 6e3a 2042 6173 6963 204d 4441  ation:.Basic.MDA
        0x0090:  7852 4545 304e 5468 434f 4445 784e 5549  xREE0NThCODExNUI
        0x00a0:  324f 5451 794d 6a4d 3652 304a 6e56 6d4a  2OTQyMjM6R0JnVmJ
        0x00b0:  334e 6a56 7a65 5739 3264 3063 3463 4767  3NjVzeW92d0c4cGg
        0x00c0:  7a56 6a46 4f51 7a5a 3056 565a 3556 324e  zVjFOQzZ0VVZ5V2N
        0x00d0:  354d 584a 5251 6b78 305a 5842 7a62 513d  5MXJRQkx0ZXBzbQ=
        0x00e0:  3d0d 0a48 6f73 743a 2072 616e 6368 6572  =..Host:.rancher
        0x00f0:  2d74 6573 742e 7069 6265 6e63 686d 6172  -test.pibenchmar
        0x0100:  6b2e 636f 6d0d 0a0d 0a                   k.com....

I would suggest putting in a PR or issue with the Lua library, as that is clearly wrong… https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.23