Ruby Net::HTTP: create a Sume bulk queue and poll to an exit code

Ruby can drive a Sume bulk queue with net/http and json alone. Create the queue with POST /v1/formats/{handle}/{slug}/bulk-runs, check for 202, then poll the receipt's status_url with a doubling gap until status is completed, and turn counts.failed plus counts.canceled into the process exit code. The script below does that in 30 lines and was run against a stub server before it was written up here.

Keep the queue id it prints. There is no endpoint that lists queues, so that line is the only record you have of which queue holds which rows.

The script

Set SUME_API_KEY, SUME_FORMAT (handle/slug) and BATCH_KEY. The key needs formats:write to create and formats:read to poll. items.json is an array of item bodies, each with at least one of instruction, input, previous_run_id or attachments.

require "json"
require "net/http"
BASE = URI(ENV.fetch("SUME_BASE", "https://api.sume.com/v1/"))
KEY = ENV.fetch("SUME_API_KEY")
def call(req)
  Net::HTTP.start(req.uri.host, req.uri.port, use_ssl: req.uri.scheme == "https", read_timeout: 30) { |h| h.request(req) }
end
def authed(req, idem = nil)
  req["Authorization"] = "Bearer #{KEY}"
  req["Content-Type"] = "application/json"
  req["Idempotency-Key"] = idem if idem
  req
end
post = authed(Net::HTTP::Post.new(BASE + "formats/#{ENV.fetch('SUME_FORMAT')}/bulk-runs"), ENV.fetch("BATCH_KEY"))
post.body = JSON.generate(concurrency: 4, items: JSON.parse(File.read("items.json")))
res = call(post)
abort "create #{res.code}: #{res.body}" unless res.code == "202"
q = JSON.parse(res.body)["data"]
puts "queue #{q['id']}"
gap = 15
until q["status"] == "completed"
  sleep gap
  gap = [gap * 2, 60].min
  res = call(authed(Net::HTTP::Get.new(URI(q["status_url"]))))
  next if %w[429 503].include?(res.code)
  abort "poll #{res.code}: #{res.body}" unless res.code == "200"
  q = JSON.parse(res.body)["data"]
end
puts q["counts"].to_json
exit(q["counts"].values_at("failed", "canceled").sum.zero? ? 0 : 1)

Why it stops where it stops

Net::HTTP does not raise on 4xx or 5xx; it hands you the response. That is why the script checks res.code itself. On create, anything other than "202" aborts with the body printed, because every create error is final: the API charges nothing for a 4xx, and retrying a 403 insufficient_scope in a loop is the most common and most expensive mistake on this surface. A 400 invalid_request names the bad item in details.index.

On poll, 429 and 503 are skipped with next, and the doubling gap does the waiting. The docs say a queue keeps draining through those. Any other poll status aborts. The queue is not canceled by that, because it is a server-side list: stopping your Ruby process only stops the poller.

Reading the finish

completed means every item is terminal, and not that every item worked. The exit status is the check that matters. When it is 1, open GET /v1/format-runs/{run_id} for each failed item. The item error is only format_run_failed, format_run_canceled, or, for a child that never started, the create-run error with run_id set to null.

If you want per-item output instead of a single code, q["items"] holds index, status, run_id and error for each row, in the order you sent them.

Two limits are worth knowing before you load a large file. The concurrency field you send is the window of children running at once, from 1 to 16, and the plan's write limit is per minute and separate from it: one bulk create counts as one write however many items it carries. Each item is checked as a single-run body, so an item over 2 MiB of input, or with more than 64 top-level input keys, fails the whole create with 400 and the index of the bad item before any queue exists.

For a rerun after a partial failure, build a new items.json from only the failed indexes and use a new BATCH_KEY. Reusing the old key with a different body returns 409 idempotency_conflict.