Most “network problems” on a server are not packets lost on a wire. They are questions answered by the wrong tool: a name that one program resolves and another says does not exist, a service that answers every request from the machine itself and none from anywhere else, a DNS server or a route that was correct when someone set it and gone after the next restart of something unrelated. Each has a precise cause, and each is found by asking the question the way the application asks it.
This journal works through those on a Rocky Linux 10 lab machine managed by NetworkManager, and every output in it was recorded there — including one assumption of the author’s that turned out to be wrong when tested, which is noted where it happened.
getent, ping and curl can disagree
with dig and host.dig’s answer and exit status correctly, and not mistake one for the other.ss alone./etc/resolv.conf.Applications do not “query DNS”. They call the C library’s resolver (getaddrinfo), which follows the
hosts: line in /etc/nsswitch.conf:
$ grep '^hosts' /etc/nsswitch.conf
hosts: files dns myhostname
files is /etc/hosts, dns is the servers in /etc/resolv.conf, myhostname answers for the
machine’s own name — in that order, first answer wins. So a name added to /etc/hosts:
$ echo "10.20.30.40 db.internal" | sudo tee -a /etc/hosts
is a real name for every program that uses the resolver — and does not exist for the DNS tools, which
talk to a DNS server directly and never read /etc/hosts:
$ getent hosts db.internal ; echo "getent=$?"
10.20.30.40 db.internal
getent=0
$ ping -c1 -W1 db.internal | head -1
PING db.internal (10.20.30.40) 56(84) bytes of data.
$ curl -s -m 2 http://db.internal/ ; echo "curl=$?"
curl=28
$ host db.internal ; echo "host=$?"
Host db.internal not found: 3(NXDOMAIN)
host=1
$ dig +short db.internal ; echo "dig=$?"
dig=0
ping and curl resolved it (curl then timed out connecting — exit 28 is a connection timeout, not a
name failure, which would be 6). host and dig said the name does not exist, because in DNS it does
not. getent hosts NAME answers the question the application asks; dig answers a narrower
question about one DNS server. Use dig when the question really is about DNS — which record a server
holds, TTLs, which server answered — and getent when the question is “what will my program connect
to”.
Note the last line: dig exits 0 for a name that does not exist. Its exit status says whether it got
an answer from a server, and NXDOMAIN is an answer. +short prints nothing and exits 0. A script that
checks dig +short NAME for success is checking that the network is up, not that the name resolves.
Read the status in the header (dig NAME | grep status:) or test for non-empty output.
/etc/resolv.conf$ cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 192.168.5.3
nameserver fec0::3
The first line is the warning. On this machine systemd-resolved is not running
(systemctl is-active systemd-resolved → inactive), and NetworkManager writes the file from the active
connection profiles — here from DHCP. A hand edit is not rejected; it is simply overwritten the next time
NetworkManager regenerates the file. Tested on the lab machine:
$ printf 'nameserver 192.168.5.99\n' | sudo tee /etc/resolv.conf
$ sudo nmcli device reapply eth0 ; cat /etc/resolv.conf
nameserver 192.168.5.99
$ sudo systemctl restart NetworkManager ; cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 192.168.5.3
nameserver fec0::3
The edit survived a reapply and vanished at the next restart of NetworkManager — and it would vanish at
a DHCP renewal or a reboot too. That is the most insidious kind of fix: it works long enough to be
forgotten. Set DNS in the profile:
$ con=$(nmcli -g GENERAL.CONNECTION device show eth0) # "cloud-init eth0" here
$ sudo nmcli connection modify "$con" ipv4.dns 192.168.5.3 ipv4.ignore-auto-dns yes \
ipv4.dns-search lab.internal
$ sudo nmcli device reapply eth0 ; cat /etc/resolv.conf
# Generated by NetworkManager
search lab.internal
nameserver 192.168.5.3
nameserver fec0::3
ipv4.ignore-auto-dns yes stops the DHCP-supplied servers being added to yours; without it you get both.
The fec0::3 line is still there because it comes from IPv6 autoconfiguration, which is a separate
setting (ipv6.ignore-auto-dns). After a reboot the file was unchanged — this configuration lives in the
profile.
$ ss -lnt | grep 860
LISTEN 0 5 0.0.0.0:8602 0.0.0.0:*
LISTEN 0 5 127.0.0.1:8601 0.0.0.0:*
The local address column is the whole story. 127.0.0.1:8601 accepts connections only to the
loopback address; 0.0.0.0:8602 accepts them on every IPv4 address the machine has. From the machine
itself:
$ curl -s -o /dev/null -m 2 -w '%{http_code}\n' http://127.0.0.1:8601/
200
$ curl -s -o /dev/null -m 2 -w '%{http_code} exit=%{exitcode}\n' http://192.168.5.15:8601/
000 exit=7
$ curl -s -o /dev/null -m 2 -w '%{http_code}\n' http://192.168.5.15:8602/
200
The loopback-bound service refused a connection to the machine’s own network address (exit 7,
connection refused) — exactly what a client on another host would get. This is the classic “it works on
the server”: every test run on the server with localhost passes. Test with the address a client uses.
Binding loopback is often deliberate — a database, a model server, anything that should be reached only
through a local proxy (see the ai-01 journal). The mistake is not binding loopback; it is not knowing you
did. And a connection from the machine to its own address travels over the loopback interface, so it
tests the binding, not the firewall: firewalld’s zone rules apply to traffic arriving on eth0 from
elsewhere (see rhcsa-04).
$ ip route
default via 192.168.5.2 dev eth0 proto dhcp src 192.168.5.15 metric 100
192.168.5.0/24 dev eth0 proto kernel scope link src 192.168.5.15 metric 100
$ ip route get 10.20.30.40
10.20.30.40 via 192.168.5.2 dev eth0 src 192.168.5.15 uid 501
ip route get answers the only question that matters — for this destination, which gateway, which
interface, which source address — using the same lookup the kernel uses for real traffic, including
policy rules and metrics you might not have spotted in the table.
A route added with ip route add is a change to the running kernel:
$ sudo ip route add 10.99.0.0/16 via 192.168.5.2
Here the author expected a NetworkManager restart to remove it, tested that, and was wrong: after
systemctl restart NetworkManager the route was still in the table — NetworkManager does not flush routes
it did not create. A reboot did remove it. A route in the connection profile survived both:
$ sudo nmcli connection modify "$con" +ipv4.routes "10.20.30.0/24 192.168.5.2"
$ sudo nmcli device reapply eth0
$ ip route | grep -E '10\.(20|99)'
10.20.30.0/24 via 192.168.5.2 dev eth0 proto static metric 100
10.99.0.0/16 via 192.168.5.2 dev eth0
…after a reboot:
10.20.30.0/24 via 192.168.5.2 dev eth0 proto static metric 100
The two lines even look different before the reboot: NetworkManager’s route carries proto static metric 100; the hand-added one has neither. That is how to spot a runtime-only route in a table before a reboot
finds it for you.
An application on this server is configured to use db.internal. A new monitoring host cannot reach the
application’s API on port 8601. On the server, the team says, “everything works”.
1. Reproduce the claim, then reproduce the client.
$ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8601/
200
$ curl -s -o /dev/null -m 2 -w '%{http_code} exit=%{exitcode}\n' http://192.168.5.15:8601/
000 exit=7
Works on loopback, refused on the address clients use.
2. Ask what is listening, and on which address.
$ ss -lntp | grep 8601
LISTEN 0 5 127.0.0.1:8601 0.0.0.0:*
Bound to loopback only. The fix is in the application’s configuration (bind 0.0.0.0 or the specific
address), and then the firewall has to allow the port — a separate step, because a local test cannot
see the firewall.
3. Meanwhile, someone “found a DNS problem”.
$ dig +short db.internal ; echo "exit=$?"
exit=0
$ host db.internal
Host db.internal not found: 3(NXDOMAIN)
Empty output with a zero exit, and NXDOMAIN. But the application does not ask DNS:
$ grep '^hosts' /etc/nsswitch.conf
hosts: files dns myhostname
$ getent hosts db.internal
10.20.30.40 db.internal
The name comes from /etc/hosts, and the application resolves it correctly. There is no DNS problem —
there is a name that exists only on this one machine, which is its own risk: move the application and
the name goes nowhere. If it should exist everywhere, it belongs in DNS.
4. Check the fixes from last week are real. The DNS server was “changed” by editing
/etc/resolv.conf, and a route to the database network was “added”:
$ head -1 /etc/resolv.conf
# Generated by NetworkManager
$ nmcli -g ipv4.dns,ipv4.ignore-auto-dns connection show "$con"
…
$ ip route | grep 10.20.30
10.20.30.0/24 via 192.168.5.2 dev eth0
No proto static on the route: it was added with ip, and the next reboot removes it. Put both in the
profile (ipv4.dns, ipv4.ignore-auto-dns, +ipv4.routes), reapply, and confirm the route now shows
proto static.
5. Prove it with the restarts that matter. Restart NetworkManager and check /etc/resolv.conf; reboot
and check ip route. On the lab machine the profile’s DNS servers and route survived both, the hand edit
did not survive the NetworkManager restart, and the ip-added route did not survive the reboot.
Testing a name with dig or host and concluding the application cannot resolve it. They bypass
/etc/hosts and NSS. getent hosts NAME resolves the way the application does.
Treating dig’s exit status as “the name exists”. It exits 0 on NXDOMAIN. Check status: in the
header, or that +short printed something.
Editing /etc/resolv.conf on a NetworkManager machine. It lasts until NetworkManager rewrites the file
— which on the lab machine was the next systemctl restart NetworkManager. Set ipv4.dns in the
profile.
Forgetting ipv4.ignore-auto-dns. Your servers are added to the DHCP ones, not substituted for them,
and which one answers depends on order.
Testing a service with curl localhost. A loopback-bound service passes that test and refuses every
real client. Test with the address a client uses, and read the local-address column of ss -lnt.
Assuming a local test proves the firewall. Traffic to the machine’s own address goes over loopback. Test from another host, or read the zone’s rules.
ip route add as a fix. It survives more than you might expect (a NetworkManager restart, on the lab
machine) and not the reboot. nmcli connection modify … +ipv4.routes, and look for proto static.
Reading the route table instead of asking. ip route get DEST applies metrics, policy rules and the
source-address choice for you.
Keeping important names in one machine’s /etc/hosts. They resolve on that machine only, and the next
server built for the same job cannot find its database.
| Symptom | Usual cause | The evidence |
|---|---|---|
dig name answers, curl name says it cannot resolve |
the name is in /etc/hosts, or NSS asks another source first or not at all |
getent hosts name; the hosts: line of /etc/nsswitch.conf |
curl name works, dig name gets NXDOMAIN |
the name exists only in /etc/hosts or a local resolver’s zone |
getent hosts name; dig @server name |
| a DNS server you set by hand is gone after a reboot or a reconnect | something else owns /etc/resolv.conf — NetworkManager or systemd-resolved |
ls -l /etc/resolv.conf; nmcli -f ipv4.dns connection show NAME |
a service answers curl localhost:PORT and times out from other machines |
it listens on 127.0.0.1, or a firewall drops the port |
ss -ltnp 'sport = :PORT'; the firewall’s rules |
| connections to one network go out the wrong interface | a more specific or lower-metric route elsewhere | ip route get ADDRESS |
a route added with ip route add is gone the next day |
it was never in the connection profile | nmcli -f ipv4.routes connection show NAME |
| “connection refused” versus a timeout | refused: something answered with a reset (nothing listens); timeout: nothing answered at all (a drop, or no route) | ss -ltn; the firewall; ip route get |
# names, the way applications resolve them
grep '^hosts' /etc/nsswitch.conf # files dns myhostname — the order
getent hosts NAME ; getent ahosts NAME # through NSS: /etc/hosts, then DNS, …
dig NAME ; dig +short NAME # DNS only — exit 0 even for NXDOMAIN
dig NAME | grep status: # NOERROR / NXDOMAIN / SERVFAIL
dig @192.168.5.3 NAME # ask one specific server
host NAME # DNS only, exit 1 on NXDOMAIN
# who configures DNS
head -1 /etc/resolv.conf # "Generated by NetworkManager" = do not edit
systemctl is-active systemd-resolved
nmcli connection modify "$con" ipv4.dns "A B" ipv4.ignore-auto-dns yes ipv4.dns-search DOMAIN
nmcli device reapply eth0
# listening
ss -lntp # local address: 127.0.0.1 = loopback only, 0.0.0.0 = all IPv4
ss -lnup # UDP
curl -m 2 http://CLIENT_FACING_ADDRESS:PORT/ # not localhost
# curl exit: 6 cannot resolve, 7 refused, 28 timeout
# routes
ip route ; ip -6 route
ip route get DEST # gateway, interface and source the kernel will use
ip route add NET via GW # runtime only (gone at reboot)
nmcli connection modify "$con" +ipv4.routes "NET GW" ; nmcli device reapply eth0
# NetworkManager's routes show "proto static metric N"; hand-added ones do not
# addresses
ip -4 -br addr ; nmcli -g GENERAL.CONNECTION device show eth0
/etc/hosts and compare getent hosts, ping -c1, dig +short
and host for it. Which ones read the file, and why?python3 -m http.server 8000 --bind 127.0.0.1, then with --bind 0.0.0.0; read ss -ltn for
each and connect from another machine or container. What does each state look like in ss?nmcli connection modify … ipv4.dns, then nmcli connection up, and read
/etc/resolv.conf before and after. Then edit the file by hand and reconnect.ip route get for an address on the local network, for 1.1.1.1, and for an address a more
specific route covers. Which interface and source address does the kernel pick each time?curl against both. Which error does each produce?man 5 nsswitch.conf, man 3 getaddrinfo — how programs resolve names.man 5 resolv.conf — the resolver’s configuration, and who is meant to write it.man 8 ss, man 8 ip-route — sockets and routes, asked of the kernel.man 1 nmcli, man 5 nm-settings-nmcli — connection profiles, and the properties that persist.A name is in /etc/hosts. ping NAME works and host NAME says NXDOMAIN. Which one tells you what an
application will do, and why do they differ?
ping— likecurland any program usinggetaddrinfo— resolves through NSS, which reads/etc/hostsfirst according to/etc/nsswitch.conf.hostanddigquery a DNS server directly and never read/etc/hosts.getent hosts NAMEis the dedicated tool for the application’s view.
A health script runs dig +short db.example.com && echo ok. What does it actually test?
That a DNS server answered.
digexits 0 even for NXDOMAIN, and+shortthen prints nothing, so the script printsokfor a name that does not exist. Check thestatus:line or test for non-empty output.
You edit /etc/resolv.conf on a NetworkManager-managed machine. How long does the change last, and
where should it be made?
Until NetworkManager regenerates the file — on the lab machine it survived
nmcli device reapplyand was overwritten by the next NetworkManager restart; DHCP renewals and reboots do the same. Setipv4.dns(and usuallyipv4.ignore-auto-dns yes) in the connection profile and reapply.
ss -lnt shows 127.0.0.1:8601. What does curl http://127.0.0.1:8601/ prove, and what do clients on
other machines get?
It proves the service works over loopback only. The socket accepts connections to 127.0.0.1, so connections to the machine’s network address are refused (curl exit 7) — which is what every remote client gets.
Why is a test from the server to its own network address not a test of the firewall?
Traffic from a machine to its own address is delivered over the loopback interface, so it never arrives on the external interface where the zone’s rules apply. It tests the service’s binding, not the firewall.
A route was added with ip route add and a route was added with nmcli connection modify +ipv4.routes. How can you tell them apart in ip route, and which survives a reboot?
The NetworkManager route shows
proto static metric 100; the hand-added one shows neither. Only the profile’s route survives a reboot. (On the lab machine theiproute even survived a NetworkManager restart — which is why restarting NetworkManager is not a test of persistence.)
What does ip route get 10.20.30.40 tell you that reading ip route might not?
The result of the kernel’s actual lookup for that destination — the gateway, the outgoing interface and the source address — after metrics, more specific routes and policy rules are applied.