It's Docker-ing Time! Part 3
Multiple IPs and Nuanced Security.
Intro and Musings
Well hello there! Doing a part three on this kind of surprised me, but as I learn new things I like to put fingers to keyboard and further document my journey in the hope that others will also find it useful. I think that this may be one of the more important posts I've made as it will highlight a nuance to the docker solution that isn’t as obvious as it should be.
So let’s start off with the use case that brought us here.
Making A Case
Ok, so. I have my one docker host at 10.0.0.10. I have about 35 containers and they all use the standard host:container convention to make the resources available. I’ve never seen a different convention, in any other compose, ever.
So at 10.0.0.10 I have WUD on port 30010 and NPM (Nginx Proxy Manager) on port 443. I have the same setup on 10.0.0.11. My eye in the sky, Uptime Kuma keeps tabs on everybody for downtime notifications.
So, lately I’m been having an issue where across vlans my firewall is having issues sending traffic to the right destinations where the FQDNs have the same IP and causing havoc with my firewall rules due to some dopey DNS cache.
For example, abs.nscriven.net and pulse.nscriven.net both go to Nginx at 10.0.0.10. So my initial solution was to build another docker host, just to proxy that IP from a unique address. A total waste of resources and a major PIA as now I need to duplicate WUD, DNS entries, Proxy entries, blah blah blah. But I did it, because my son was on me because he couldn’t access these resources externally. (Home Enterprise, remember ;) )
So I did this as a stopgap measure. I told my buddy Alex (who I work with) about this and he told me, “why not just use a second IP on your first host and use that for your conflicting containers?”
Me: /dumb look…”I didn’t know I could do that.”
IP Address Value Meal
Ok, so adding the secondary address on the same vlan is really easy it turns out. For Ubuntu, you edit netplan, and add a second address under the first and then save and “apply netplan”. They should respond to a ping right away.
So my thought was, I’ll decom 10.0.0.11 and add that IP to the 1st host, light up Nginx and profit. I won’t have to redo firewall rules, DNS, etc. So the new design looks like:
Note: There’s more other ways to this with macvlans, but it’s complicated and my eyes legit gloss over when reading about it.
So now I have this plan, right? How do I get it done? Well it turns out you can bind containers to IPs! The convention is IP: host port: container port.
And it worked! Now it’s important to note in this fashion I did not make a new bridge, this is a 1:1 mapping to the new IP. No other containers will use this combination. It’s a very specific use case for a very specific problem. I replicated it for another service for the same issue to great success.
System Of A (Not Going) Down
So…my task completed I waited for Kuma to send my glorious down alert for WUD, since that container no longer exists on 10.0.0.11 as a host. And….it didn’t arrive. I check Kuma, and it is willfully, defiantly, mockingly GREEN.
This didn’t make sense…WUD isn’t configured on .11, just .10! Why is it still up?! Then something hit me based off some other conversations I had seen on Redditt recently. And it has to do with the way docker does networking, and the arguably sloppy way developers make compose files.
The Tyranny Of The Default
So, if you have one IP on your box I’d say what follows isn’t likely a concern but the education is important. It is disturbingly easy to be so enamored with a technology like Docker and not realize you’re glossing over important nuance because it feels like magic when it works. So let’s shine some light here.
When you see host:container, what you’re saying is that those ports can be reached BY ANY IP ON THE BOX. To put it another way:0.0.0.0:host:container
Now, I’ve seen that stated elsewhere and scoffed “why does that matter?” I WANT the port to be reached. That’s the whole point.” And the scoffing wasn’t just me, many others were making this flawed logical assumption.
I work in the security space, and once I saw this in the flesh all of a sudden everything made sense. What was happening was WUD was now reachable over port 30110 from BOTH IPs! I call this “unintended access” in my opinion there is no greater sin than unintended access, and even worse when you did it unknowingly.
If I wasn’t such a control freak and treated my network as pseudo-Enterprise I would have NEVER have known about it. Here’s what this now looked like.
The plan was in no way, shape or form intended for this type of access. But there it was. Now, I’ve seen people say:
Put it behind a proxy, who cares
Don’t you have a firewall?
I don’t like these arguments. I have both. People forget L2 is a thing, and segmentation at L2 is complex and expensive. If I am exposing anything, it must be intentional, and the risk/tradeoff fully known and understood. Security is in layers and it starts as close to the resource as possible and radiates outwards with other controls.
So this must be fixed. Luckily it’s simple.
The Fix Is In
Ok so the cardinal rule here is, if your box has one IP, host:container is likely fine. (I’ve seen arguments that 127.0.0.1 can be a concern but I’d need to see a use case) But the second you add any network resource every.container.forevery.port should be tied to an IP, even the first IP on the box.
So for WUD I updated my compose to 10.0.0.10:30010:3000. And then Kuma gave me the down alert I was looking for. I then repeated this for every container on that box to ensure access I wanted was intended and narrow.
Conclusion
So I hope I’ve properly communicated the use case, resulting issue, and fix. This is one of those things you find by accident; or are knowledgeable about due to time and study because you’re an expert. Think if you’re running a container in a VPS or shared environment that has more than one IP and you’re unwittingly exposing a service to the outside world in a way you didn’t intend or protect for. For me, it’s a re-dedication to learning more about the solutions I use in my Home Enterprise and paying more heed to peers that may be trying to say something important.
Until next time.