Prometheus Target DOWN: 'context deadline exceeded' and Other Causes

You open Status → Targets in Prometheus and a target is red:

State: DOWN
Error: Get "http://app:9100/metrics": context deadline exceeded

DOWN means Prometheus tried to scrape the target and the HTTP request failed. Prometheus is working fine — it's telling you it couldn't reach or read the endpoint. The Error string is the whole diagnosis. Here are the ones you'll actually see.

'connection refused' — nothing is listening

Error: Get "http://app:9100/metrics": connection refused

The host is reachable but no process is listening on that port. Either the exporter/app isn't running, or it's on a different port than your scrape config claims. Check the target is up and exposing metrics:

curl http://app:9100/metrics    # run this from where Prometheus lives

If curl also refuses, it's not a Prometheus problem — start the exporter or fix the port.

'context deadline exceeded' — the scrape timed out

The target accepted the connection but didn't return /metrics before scrape_timeout. Causes:

• The endpoint is genuinely slow (huge metric set, overloaded app).
• A firewall is silently dropping packets (you'll see a timeout, not a refusal).
• scrape_timeout is shorter than scrape_interval allows.

Raise the timeout in your scrape config, and time the endpoint by hand:

scrape_configs:
  - job_name: app
    scrape_interval: 15s
    scrape_timeout: 10s        # must be <= scrape_interval
    static_configs:
      - targets: ['app:9100']

'no such host' — DNS can't resolve the target

Error: Get "http://app:9100/metrics": dial tcp: lookup app: no such host

The hostname doesn't resolve. In Docker/Kubernetes this usually means Prometheus and the target aren't on the same network, or you used a name that only exists elsewhere. Use a name reachable from the Prometheus container — service name on a shared Docker network, or service.namespace in Kubernetes.

'404 Not Found' or '400' — wrong path or scheme

The scrape connected but the endpoint isn't where Prometheus looked. Most exporters serve /metrics; if yours uses a different path, set metrics_path. If the target is HTTPS, set the scheme:

    metrics_path: /actuator/prometheus
    scheme: https

Always: reload and re-read the page

After editing prometheus.yml, reload config and check /targets again:

curl -X POST http://localhost:9090/-/reload    # needs --web.enable-lifecycle

The checklist

1. Read the exact Error on Status → Targets — it names the cause.
2. connection refused → exporter down or wrong port; curl /metrics to confirm.
3. context deadline exceeded → slow endpoint or firewall; raise scrape_timeout.
4. no such host → DNS/network; use a name reachable from Prometheus itself.
5. 404 → set the right metrics_path (and scheme: https if needed).
6. Reload config and recheck the Targets page.

Frequently Asked Questions

What does 'context deadline exceeded' mean in Prometheus?

It's a scrape timeout — Prometheus connected to the target but didn't get the /metrics response before scrape_timeout elapsed. Usually the endpoint is slow or a firewall is dropping packets. Raise the timeout and time the endpoint with curl.

Why is my target UP in curl but DOWN in Prometheus?

Because you're curling from a different place than Prometheus scrapes from. Run curl from the Prometheus host/container — in Docker or Kubernetes the target name often resolves for you but not for Prometheus unless they share a network.

Where do I see why a Prometheus target is down?

On the web UI under Status → Targets. Each target shows its State and, when DOWN, an Error string that states the exact reason (connection refused, timeout, DNS failure, 404, etc.).