Resources

A resource is something outside the pipeline that has versions: a git branch, a queue of pull requests, a build artifact, a counter in a file. A resource type is the code that knows how to talk to one.

Every example on this page (and every other page) is a complete pipeline, verified by the test suite. Blocks that need the network or real credentials say so; everything else runs as shown.

resources:
- name: repo          # the artifact directory a get creates, and the name steps use
  type: git           # which resource type
  source:             # configuration for that type — free-form, type-specific
    uri: https://github.com/jtarchie/ci.git
    branch: main

jobs:
- name: build
  plan:
  - get: repo         # fetch it — the contents land in ./repo
  - task: compile
    inputs: [repo]
    run: cd repo && go build ./...

git ships with steps, so that pipeline needs no resource_types: block. For anything else, you write the type yourself.

The built-in git type

It fetches the exact commit the plan pinned, shallowly, so a branch that moves mid-run still gives you the version that was planned. It has no out: — put: repo against it is a load error, because what "publish" means (which branch, which credentials, force or not) is a decision only you can make. Write your own type for that.

The built-in slack-mentions and slack-reply types

Two more built-ins, both expression-backed — Slack is a JSON HTTP API and nothing else, so there is no container and no curl/jq dependency to carry. slack-mentions is get-only: every unanswered @mention of the bot in a channel, plus every message in a 1:1 DM (no @mention required there — nobody types one in a 1:1 chat), oldest first, as a {channel, ts, thread_ts} version — ts is the message that named the bot, thread_ts the thread it lives in (the same value, for a top-level message). slack-reply is put-only: posts a message, threaded or top-level.

Cold start does not mean a backlog: like every resource, the first-ever check of a freshly-deployed slack-mentions records everything it finds and answers only the newest of it — see Version history below. That rule is a steps web behavior; steps run has no persisted cursor, so every steps run check is a first check.

A cursorless check is also bounded in what it asks Slack. Finding a mention written as a reply means one conversations.replies call per thread, and the cursor is what normally keeps that set small — no cursor means every thread that has ever been replied to qualifies. On a workspace with more threads than the rate limit allows calls for, that check gets throttled, and a throttled check reports no version, which leaves no cursor for the next one: a watcher that can never run once. So a check with no cursor looks at the five most recently active threads and stops.

Under steps web that costs nothing that is ever built: everything below the newest version a first check reports is marked already-taken, and the cursor it leaves behind lifts the bound for every poll after it. Under steps run the bound never lifts, because no cursor is ever persisted — a mention written inside the sixth-most-recently-active thread is not delivered. Use steps web for a Slack bot; steps run is for a one-shot answer to whatever is most recent.

resources:
- name: mentions
  type: slack-mentions
  source:
    channels: []   # optional; [] (the default) is every channel the bot is in
    limit: 200      # optional; per-channel messages fetched per check, default 200

- name: reply
  type: slack-reply
  source: {}

# A second bot, in the same pipeline, answering as someone else: env: adds
# SECOND_BOT_TOKEN to just THIS resource's own allow-list (the type's own
# env: still only names SLACK_BOT_TOKEN), and source.token_env picks it.
- name: reply-as-support-bot
  type: slack-reply
  env: [SECOND_BOT_TOKEN]
  source:
    token_env: SECOND_BOT_TOKEN

jobs:
- name: answer-mention
  plan:
  - get: mentions
    trigger: true
    version: every    # answer every mention found, not just the newest
  - task: address
    inputs: [mentions]
    outputs: [thread, answer]
    run: |
      set -eu
      grep -o '"channel": *"[^"]*"' mentions/version.json | cut -d'"' -f4 > thread/channel
      grep -o '"thread_ts": *"[^"]*"' mentions/version.json | cut -d'"' -f4 > thread/ts
      echo "got it, working on it" > answer/reply.md
  - put: reply
    inputs: [thread, answer]

Both need SLACK_BOT_TOKEN (a bot token, xoxb-) in the environment, for an app with chat:write, channels:history, channels:read, im:history and im:read (plus groups:history/groups:read for private channels), installed to the workspace and invited to every channel it should watch or post to. Membership is what grants both reading history and appearing in users.conversations, so /invite is the subscribe action — app_mentions:read is not needed, since this polls rather than using the Events API.

1:1 DMs are always watched too, with no @mention required — there is no source: field to turn this off. Group DMs (mpim) are not watched at all; a group is closer to a channel (several humans, bot is one more party) than a 1:1, so it keeps the explicit-mention rule instead.

token_env alone isn't enough to widen what a resource can read — env() only sees names its resource TYPE already declares (both types declare SLACK_BOT_TOKEN, shared by every resource of that type), which is what makes it safe for a shared, possibly-external type to hand-in-hand with any expr type at all. A resource naming a different token also needs env: on the resource itself to add that name to its own allow-list — env: and source: together, as in reply-as-support-bot above. Naming token_env without the matching env: entry is a run-time error (env(...): not in this resource type's env:), not a silent fall-back to SLACK_BOT_TOKEN. (env: on a resource only means something for an expr- or shell-backed type — an mcp-backed type authenticates via its mcp_servers: entry and rejects env: at load time.)

A mention inside a thread arrives with its thread. mentions/thread.json is the whole conversation the mention was written in (Slack's conversations.replies payload: parent first, then replies), fetched by thread_ts — asking Slack for a reply's ts answers with that one message and nothing around it, which is an agent being handed a question with no context. Post the answer back with thread_ts too, as the example above does: a reply's ts is not a thread id.

A mention inside a thread counts, and limit: is what decides whether it can be found. Slack's conversations.history returns top-level messages only, and a reply does not change its parent's ts — so the window the check reads channel history over is limit messages, deliberately wider than the cursor, and the cursor decides only what inside that window is new. A thread whose parent carries a latest_reply newer than the cursor is read; every other thread costs nothing.

Two known gaps remain, both bounded by limit: and both needing Slack pagination to close:

• A thread whose parent has scrolled past limit messages of its channel's history is invisible, however active the thread still is. How long that takes is how chatty the channel is; raising limit: buys more room, but only as much as Slack allows: since 2025-05-29 a bot distributed outside the Marketplace is capped at 15 messages per request and one request a minute, whatever limit: says. An internal app built for a single workspace — the usual case for a self-hosted pipeline — is excluded from that cap and honors limit: as written.
• mentions/thread.json is truncated at its newest end for a thread longer than 1000 messages — Slack returns a thread oldest-first — so the mention itself can be missing from a very long thread.
• More than limit new top-level messages in one channel between two checks lose the overflow permanently, not just delayed — the cursor advances to the newest ts seen anywhere, so whatever limit cut off now sits below the new cursor and is never asked for again.

slack-reply's put: reads its message from files an upstream step writes, not params: — file() takes what inputs: put on disk directly, so a reply containing backticks or $(…) is data, never something a shell might run:

There is deliberately no check:/in: on slack-reply and no out: on slack-mentions — get: reply or put: mentions are both load errors, the same rule git's missing out: follows.

Writing a resource type

A resource type is three shell commands. (For a resource that is a JSON HTTP API and nothing else, there is a second way to write them — see expression resource types, which trades containers and binary artifacts for concurrent HTTP and no dependency on curl/jq.) Each is a template and each runs sh -c. This one is self-contained, so it runs anywhere:

resource_types:
- name: greetings
  # image: alpine:3   # optional — run these in a container instead of on the host
  config:
    check: |
      printf '[{"word": "hello"}, {"word": "hola"}]'
    in: |
      echo {{ .version.word | shellquote }} > word.txt

resources:
- name: greeting
  type: greetings
  source: {}

jobs:
- name: speak
  plan:
  - get: greeting
  - task: shout
    inputs: [greeting]
    run: tr a-z A-Z < greeting/word.txt
    assert:
      stdout: HOLA           # check printed oldest-first, so the LATEST is "hola"
  assert:
    execution: [greeting, shout]
    outcome: succeeded

check — what versions exist?

Runs when a plan is built, and on every steps web poll.

• Sees: {{ .source }} and {{ .version }} — the last version this pipeline recorded for the resource. See the cursor below.

• Must print: a JSON array of version objects to stdout, oldest first. A version object is a flat map of strings — {"ref": "abc123"}, {"number": "87"}. The whole object identifies the version; steps never interprets the fields.

• Empty array means "no versions yet". Any version mode other than every fails the step with no versions available. Under version: every the job exits 0 having built nothing, and says which kind of nothing it was:

  • get: <name> cannot build; no versions exist for: <names> — an input has never had a version at all, so no set can be assembled. The named resources are what to go look at.
  • get: <name> has no new versions; all N already taken — the steady state: everything this check reports has been built already.
  • get: <name> returned no versions; the N step(s) after it did not run — the check came back empty and there is no history to fall back on, so that much plan was dropped.
  • get: <name> returned no versions; nothing was fetched — the same, for a get inside a build: the rest of the plan still runs, without that artifact.

  These say a check came back empty — they cannot say why, so a type should still fail loudly (exit non-zero) when it can't answer, rather than printing nothing.

• Exit non-zero to fail the step.

[{"ref": "9fceb02"}, {"ref": "d7b22a6"}]

The check cursor

{{ .version }} is the newest version the last successful check reported — Concourse calls it the current version. It exists so a check can ask its API for what it has not seen instead of guessing a window:

# a guess, and the only thing between you and both failure modes below
check: |
  curl -sS ... --data-urlencode 'limit=20' https://api.example.com/messages

# ask for exactly what we haven't seen
check: |
  curl -sS ... --data-urlencode 'since={{ index .version "ts" | default "0" }}' \
               --data-urlencode 'limit=200' https://api.example.com/messages

Guess too small and items scroll past during a busy period — and while history means a version steps already recorded is not lost, one it never saw at all cannot be recovered by anything. Guess anything at all and a cold start sees a backlog it must not answer — it builds only the newest of what it finds.

Three things to know:

• Spell it {{ index .version "ts" | default "0" }}, not {{ .version.ts }}. On the first-ever check there is no cursor and .version is an empty map; templates render with missingkey=error, so the bare form fails that first poll. This is the same shape an optional source: field or get params: already uses.
• The cursor belongs to steps web alone, which both advances it and is the only thing that reads it. A steps run or steps plan renders {{ .version }} as an empty map even when a watcher has been polling for weeks — a manual run asks what exists, not what is new.
• A check that ignores .version keeps working exactly as before. The cursor narrows what a check asks for; it is not a filter steps applies to the answer, and it is not what decides which versions a job builds — that is history.

Version history

steps remembers every version it has seen of a resource, in the order it first saw them. That record is what a triggered job actually builds from — it does not re-run check for the versions it was triggered for.

The reason is that a cursor-driven check cannot be asked twice. The second answer is different, because the first answer moved the cursor: a job re-deriving its own versions would ask "what is new since the versions I was just handed" and correctly get nothing. A lookup is repeatable, so plan time and run time agree without anything being passed between them.

History is also what makes a version recoverable. Before it, whatever check returned right now was the whole universe — a version that scrolled out of the window while nothing was watching was gone, and no amount of cursor bookkeeping could bring it back.

Three things follow:

• A resource nothing has polled has no history, so a get against it runs its own check — every steps run, and every get: beside a triggered one.
• A cold start does not become a backlog. The first check of a resource records everything it reports, marks everything BELOW the newest as already taken, and triggers once on the newest — so a watcher pointed at twenty existing items builds the twentieth and answers none of the nineteen. One build, not twenty (which is what the marking is for) and not zero (which left a slowly-changing resource unbuilt indefinitely, and made deleting the state database re-arm that silence). version: every does not change this: the fan-out is over UNCONSUMED versions, and a cold start leaves exactly one unconsumed. (A job added to the pipeline later has no such marking, and its first trigger will see whatever history holds.)
• Pruning is not free. A version dropped from history takes its green record with it, so a passed: gate can no longer clear for it. That is correct — a version out of history cannot be built — but it means a limit set below what a slow downstream job needs will hold that job back.

How much to remember

defaults.version_history: caps it per resource, keeping the newest (0 keeps everything, as every limit here does):

# The cap itself is not observable in one run — internal/store/sqlite's tests measure
# the pruning. This example pins only that the field loads and a capped
# resource still fetches normally.
defaults:
  # A git branch produces a version per push; a chat feed one per message.
  # The right number is a property of what you watch.
  version_history: 50

resource_types:
- name: counter
  config:
    check: |
      printf '[{"n": "1"}, {"n": "2"}]'
    in: echo {{ .version.n | shellquote }} > n.txt

resources:
- name: ticks
  type: counter
  source: {}

jobs:
- name: build
  plan:
  - get: ticks
  - task: show
    inputs: [ticks]
    run: cat ticks/n.txt
    assert:
      stdout: "2"
  assert:
    execution: [ticks, show]
    outcome: succeeded

--version-history sets a default for a pipeline that does not; when neither says, steps keeps 1000. Whatever the limit, the newest versions are the ones kept.

resource_types:
- name: since-cursor
  config:
    # A real type would send this to an API. Here it just reports what it was
    # handed, which is the part worth seeing: on a fresh run there is no
    # cursor, so the default is what the check gets.
    check: |
      printf '[{"seen": "%s"}]' '{{ index .version "ts" | default "0" }}'
    in: echo {{ .version.seen | shellquote }} > seen.txt

resources:
- name: feed
  type: since-cursor
  source: {}

jobs:
- name: poll
  plan:
  - get: feed
  - task: show
    inputs: [feed]
    run: cat feed/seen.txt
    assert:
      stdout: "0"          # nothing recorded yet, so the check saw the default
  assert:
    execution: [feed, show]
    outcome: succeeded

in — fetch one version

Runs when a get step executes.

• Sees: {{ .source }}, {{ .version }} (one object from check's array), and {{ .params }} (the get step's params:).
• Working directory is the artifact directory itself, already created and empty. Write the contents there — ., not a subdirectory named after the resource.
• Exit non-zero to fail the step.

params: on a get is how a resource is told how to fetch, as opposed to source:, which says what to fetch. The distinction matters because source: belongs to the resource and params: belongs to the step, so one resource can be fetched differently by different jobs without being declared twice:

resource_types:
- name: notes
  config:
    check: |
      printf '[{"ref": "v1"}]'
    in: |
      head -n {{ index .params "lines" | default "100" }} <<'EOF' > notes.txt
      first line
      second line
      EOF

resources:
- name: log
  type: notes
  source: {}

jobs:
- name: quick
  plan:
  - get: log
    params: { lines: 1 }     # this job fetches a truncated view
  - task: show
    inputs: [log]
    run: wc -l < log/notes.txt
    assert:
      stdout: "1"            # the param reached in:
  assert:
    execution: [log, show]
    outcome: succeeded
- name: full
  plan:
  - get: log                 # same resource, whole thing
  - task: show
    inputs: [log]
    run: wc -l < log/notes.txt
    assert:
      stdout: "2"            # no param, so the default won — same resource, other view
  assert:
    execution: [log, show]
    outcome: succeeded

assert:
  execution: [quick, full]

Optional params take the same shape as an optional source: field (see Shell safety below). Templates render with missingkey=error, so a bare {{ .params.lines }} makes lines mandatory on every get of that type; {{ index .params "lines" | default "100" }} works on an absent key and on a get with no params: block at all.

Params change the fetch, so they change the hash. Two gets of one version differing in params: are two different fetches: they get distinct cache entries and neither is reused for the other. A get with no params: hashes exactly as it did before the field existed, so adding this to a pipeline invalidates nothing that does not use it.

The fetched directory is named after the get:, so get: log puts it in log/, and later steps read log/.... See workspace.md for what a step can and can't see.

out — publish something

Runs when a put step executes. Optional: a type with no out: is read-only, and a put: against it is rejected at load time rather than silently doing nothing.

• Sees: {{ .source }} and {{ .params }} (the put step's params:).
• Working directory is the put step's read view, composed from its inputs:.
• May print a single JSON object — the version it produced. Printing nothing is fine and not an error.

A put publishes; it does not fetch

A put step runs out: and nothing else — there is no implicit get afterward, so a put produces no artifact. (Concourse fetches the produced version automatically; steps deliberately does not: an artifact appearing in the build that no step declared is exactly the kind of ambient data flow this DSL rejects.) A plan that wants the resource's contents after a put writes the fetch it means:

resource_types:
- name: release
  config:
    check: |
      printf '[{"ref": "v1.4.1"}]'
    in: echo {{ .version.ref | shellquote }} > ref
    out: |
      cat notes/summary.txt        # "publish" the summary an earlier step wrote
      printf '{"ref": "v1.4.2"}'

resources:
- name: releases
  type: release
  source: {}

jobs:
- name: publish
  plan:
  - task: summarize
    outputs: [notes]
    run: echo 'what changed' > notes/summary.txt
  - put: releases              # out: publishes and prints v1.4.2
    inputs: [notes]
  - get: releases              # fetch the resource, explicitly
  - task: verify
    inputs: [releases]
    run: cat releases/ref
    assert:
      # check's answer, NOT the v1.4.2 the put just printed — the get was
      # pinned when the plan was built, before out: ran. See below.
      stdout: v1.4.1
  assert:
    execution: [summarize, releases, releases, verify]   # put, then get — two entries
    outcome: succeeded

The explicit get fetches the version check reported when the plan was built — check runs once, before any step, so the version the put publishes mid-run is not what the same run's get fetches (the example above pins exactly that: out: prints v1.4.2, the get still fetches v1.4.1). The version a put prints is recorded with the run; it reaches gets in later runs, once a check has reported it — a downstream job triggered on the resource is the plan shape that consumes what a put published. A put whose output nothing reads simply has no get after it.

Shell safety

Anything interpolated into a command is text substitution, so quote it:

check: git ls-remote {{ .source.uri | shellquote }}     # good
check: git ls-remote {{ .source.uri }}                  # a uri with a space or ; breaks or worse

shellquote renders a value as one safely-quoted shell word. Use it for every {{ }} that reaches a command. See templating.md.

Templates render with missingkey=error, so reading an optional field that wasn't set fails the render. Ask for optional fields in a way that can answer "nothing":

{{ index .source "branch" | default "HEAD" }}     # optional
{{ .source.uri }}                                 # required — failing is correct

version: on a get step

By default a get fetches the latest version check reported. version: every runs the rest of the plan once per version, and a mapping pins one exact version:

resource_types:
- name: builds
  config:
    check: |
      printf '[{"number": "87"}, {"number": "88"}]'
    in: echo {{ .version.number | shellquote }} > number.txt

resources:
- name: build
  type: builds
  source: {}

jobs:
- name: latest-only
  plan:
  - get: build                     # default: "88", the newest
  - task: show
    inputs: [build]
    run: cat build/number.txt
    assert:
      stdout: "88"
  assert:
    execution: [build, show]
    outcome: succeeded
- name: each-in-turn
  plan:
  - get: build
    version: every                 # the rest of the plan runs per version
  - task: show
    inputs: [build]
    run: cat build/number.txt      # no stdout assert: this runs once per version
  assert:
    execution: [build, show, build, show]   # the fan-out, one pass per version
    outcome: succeeded
- name: pinned
  plan:
  - get: build
    version: { number: "87" }      # exactly this one
  - task: show
    inputs: [build]
    run: cat build/number.txt
    assert:
      stdout: "87"                 # the pin won over the newer version
  assert:
    execution: [build, show]
    outcome: succeeded

assert:
  execution: [latest-only, each-in-turn, pinned]

Under every, a failing version does not stop the remaining ones from being attempted.

every takes each version once

A check reports what exists, not what is new — the same twenty Slack messages, the same page of builds, on every poll. So every remembers: once a version's fan-out has succeeded, that version is not taken again, and a later run fans out only over what is left. Without that, a plan ending in a put: or an agent: — the two steps the cache deliberately never skips, because their worth is an effect rather than an artifact — repeats every effect it has ever performed each time anything new shows up.

• Recorded per (job, resource), so another job reading the same resource keeps its own place.

• A version is taken when its build STARTS, not when it succeeds — so a version whose build failed is not retried on the next run. This is Concourse's rule (NextEveryVersion reads the versions a build was created with and never looks at build status), and it is what stops one bad input failing forever, on every trigger, with an agent's bill attached. Re-running it is a deliberate act: --force, --resume, or a new version.

• --force ignores it, along with every other piece of persisted state — which means it re-runs versions already taken, effects included. It still records what it took, so the ordinary run after a forced one does not repeat that work a second time.

• It suppresses; history is what resurrects. The versions a job may take are the ones steps has recorded, not only the ones check returns right now — so a version that scrolled out of the window while nothing was watching is still built. What history does not hold, nothing can recover: a version pruned by version_history:, or one from before steps first checked the resource.

• Any top-level get: in a plan may say every — each keeps its own cursor. Anywhere else (inside a hook, or a branch of in_parallel:/do:/try:) the get runs within a build whose versions are already decided, so every there is a load error rather than a field that is accepted and ignored. Two every gets on the same resource are a load error too — one cursor cannot serve both.

• A second get of the same resource keeps its own version. get: code, version: every beside get: baseline, resource: code, version: {ref: "v1"} fans over code while baseline stays pinned, which is how a plan diffs what just arrived against a fixed point.

steps plan reads the same record, so it lists only the versions a run would actually take.

Several every gets: input sets

When more than one get says every, a run resolves input sets, Concourse's model: each every get advances one step per set through its own unbuilt versions, in lockstep with its siblings, and one build runs per set. A get whose versions run out holds at the newest version it has already covered while the others keep moving. There is no cross product — 3 new versions on one input and 2 on another mean three builds, not six.

The hold rule is what makes the steady state right, not just the burst: updates rarely arrive in matched pairs. config moving alone builds (code@held, config@new); code catching up later builds (code@new, config@held).

resource_types:
- name: builds
  config:
    check: |
      printf '[{"number": "87"}, {"number": "88"}, {"number": "89"}]'
    in: echo {{ .version.number | shellquote }} > number.txt
- name: configs
  config:
    check: |
      printf '[{"rev": "5"}, {"rev": "6"}]'
    in: echo {{ .version.rev | shellquote }} > rev.txt

resources:
- name: build
  type: builds
  source: {}
- name: conf
  type: configs
  source: {}

jobs:
- name: pairwise
  plan:
  - get: build
    version: every
  - get: conf
    version: every
  - task: show
    inputs: [build, conf]
    # The case pins the pairing, not just the count: a cross product would
    # build (87,6), (88,5) or (89,5) and fail here.
    run: |
      pair="$(cat build/number.txt)-$(cat conf/rev.txt)"
      case "$pair" in
        87-5|88-6|89-6) echo "took $pair" ;;
        *) echo "unexpected pair $pair"; exit 1 ;;
      esac
    assert:
      stdout: took
  assert:
    # Three sets, not six: (87,5), (88,6), then conf is exhausted and
    # HOLDS at rev 6 while build keeps moving — (89,6).
    execution: [build, conf, show, build, conf, show, build, conf, show]
    outcome: succeeded

assert:
  execution: [pairwise]

trigger: true

Marks a get as something steps web should poll. When its version changes, the jobs containing it run automatically. Valid only on get steps — setting it anywhere else is a load-time error.

resource_types:
- name: ticker
  config:
    check: |
      printf '[{"tick": "1"}]'
    in: echo {{ .version.tick | shellquote }} > tick.txt

resources:
- name: clock
  type: ticker
  source: {}

jobs:
- name: on-change
  plan:
  - get: clock
    trigger: true      # steps web polls this; a new version runs the job
  - task: react
    inputs: [clock]
    run: cat clock/tick.txt
    assert:
      stdout: "1"
  assert:
    execution: [clock, react]
    outcome: succeeded

See infra.md for the watch loop, webhooks, and cross-job triggering. Gating a get on upstream jobs — Concourse's passed: — is there too: infra.md#passed.

MCP-backed types

A resource type can call an MCP server instead of running shell commands — the same check/in/out roles, as tool calls. See mcp.md.

Checking your work

steps validate pipeline.yml answers "does it parse and hang together"; steps plan pipeline.yml runs check and shows what would be fetched vs cached — the fastest way to see whether a check you just wrote returns what you expect, since it resolves versions without running the rest of the job.