How do I run one job on many services?

What is Router?

A job is something you want done, such as "pay out" or "send a text". Its services are the connected APIs and tool servers that can do it. You send the job one request. Router chooses a service by your rules, its priority and how it has been doing, sends the request in that service's own words, and gives you back one kind of answer whichever service did the work.

Every run is recorded with a sentence that says what happened and why, and one record per service it considered. Rules decide whether a run may happen at all, and the same request with the same Idempotency-Key never runs twice.

Can I try Router with no other accounts?

Yes. Cadreen has a test service built in, router_test_service. It does nothing outside Cadreen, and you choose what each one of your providers looks like and does with its params: shape a or b (two services that want different requests), and behavior succeed, refuse_before_accepting, rate_limit, time_out (it does the work, then goes quiet), time_out_before_taking (it goes quiet without taking the work), fail_after_accepting or wait. A workspace needs a rule before anything runs (see what a first run needs). Make a job with two test services, the first one refusing. The two shapes want different requests, so a request that both accept carries the fields of both (amount, currency and account for a; value_minor, ccy and account_ref for b):

POST /api/v1/cadreen/routing/groups/{id}/providers
{ "tool_id": "router_test_service", "name": "Test A", "priority": 1,
  "params": { "shape": "a", "behavior": "refuse_before_accepting", "service_name": "Test A" },
  "side_effect_class": "effectful", "equivalence_class": "test", "idempotency_support": "body_param" }

Add a second one the same way, Test B, with shape: "b" and behavior: "succeed". The service itself confirms that it passes the repeat-protection key on, so nobody has to accept a risk for Router to switch. Run the job as a test with a request both accept:

POST /api/v1/cadreen/routing/groups/{id}/execute
Idempotency-Key: try-1
{ "test_run": true,
  "input": { "amount": 5, "currency": "USD", "account": "acct_1",
    "value_minor": 500, "ccy": "usd", "account_ref": "acct_1" } }

test_run: true says the run is a test. A test service takes only test runs: it does nothing real, so its answer is a simulation, and Router never sends it real work. A run with no test_run (or false) is real work. Real work is never sent to a test service: the test service is skipped (test_service_real_work) and Router goes on to the next service, so a job can hold your real API beside the test services and each gets only what it should. A test run is the other way round: it is sent only to test services, and every real service is skipped (real_service_test_run). If a job has only test services, real work is refused with 409 only_test_services and nothing is sent: "This job has only test services, so real work can't run here. Add a real service, or use Run it to try it." The record of a test run says so (test_run: true), and so do its receipt and its webhook events. Anyone who can run a job may send test_run; it can only keep a run away from real services. A value that isn't true or false is 400 invalid_request. The same field works on executeByName and on the previews (dry-run, dryRunByName).

Router moves the work to Test B, because Test A refused before it accepted anything. Change Test A to time_out (PUT /api/v1/cadreen/routing/groups/{id}/providers/{provider_id}/params) and run again, as a test, with a new key: Router stops instead, because it can't know whether Test A did the work, and Test B is never sent it.

To send one request instead, give the job a shape (amount, currency and account), tell it what its amounts are counted in, and say how each test service speaks it. The unit matters for Test B, which wants whole minor units: with no unit, Router skips Test B, and with minor a request of 12.5 is refused, because 12.5 is not a whole number of minor units. Counted in major, 12.5 is 1250:

PUT /api/v1/cadreen/routing/groups/{id}/amount-unit
{ "amount_unit": "major" }

Test A takes the request as it is, with the currency in upper case, and its answer already has the words Router uses:

PUT /api/v1/cadreen/routing/groups/{id}/providers/{provider_id}/mapping
{ "mapping": {
    "input": [ { "to": "amount", "from": "amount" },
               { "to": "currency", "from": "currency", "transform": "upper" },
               { "to": "account", "from": "account" } ],
    "result": { "reference": { "path": "id" },
                "status": { "path": "status", "values": { "succeeded": "succeeded", "failed": "failed", "pending": "pending" } } } } }

Test B is mapped the same way to its own names: value_minor from amount with the change amount_minor, ccy from currency with lower, and account_ref from account; its answer is read from transfer.ref and transfer.state, where COMPLETED means succeeded, REJECTED means failed and PROCESSING means pending. Then you send { "amount": 12.5, "currency": "USD", "account": "acct_1" }, Test A is sent amount, currency and account as they are (the currency in upper case), and Test B is sent value_minor: 1250, ccy: "usd" and account_ref. Either way the answer comes back as status and reference.

The test service keeps a record of every call it received. It is how you answer "did Test A do the work?":

GET /api/v1/cadreen/routing/test-service/calls

Each call says which service, the repeat-protection key it got, the request as it arrived, and whether it did the work. POST /api/v1/cadreen/routing/test-service/reset clears it for your workspace.

Params are shown in plain text every time the provider is read, to everyone who can read the job. Never put a credential in them: it goes in the service's connection.

The record of calls holds the requests as they arrived, and anyone or any key that can read your workspace's jobs can read it: send the test service only test data. Clearing it needs an admin, or a key with routing.setup. The service is never offered to Router's matching, the planner or GET /tools; you name it, as above.

The record and the keys the service has seen are kept in memory by each server process, and for a limited time and number of calls. With more than one server running, a call may reach a server that didn't run it, and the record you read may be another server's.

What does a first run need?

A job, at least one service in it, and a rule that lets it run. A workspace with no rules refuses to route (no_rules_yet). A workspace admin can choose "run unless a rule says no":

POST /api/v1/cadreen/routing/starter-rules
{ "choice": "run_unless_a_rule_says_no" }

Then run the job. The same Idempotency-Key never runs the work twice:

POST /api/v1/cadreen/routing/groups/{id}/execute
Idempotency-Key: order-1
{ "input": { "amount": 5 } }

A run needs no operation of its own: the Idempotency-Key makes one, so sending the same request with the same key again does nothing twice. If that run sent nothing to any service (a rule refused it, nothing could be tried), fix what stopped it and send the same request with the same key: it runs. You can also pass operation_id, or run it from a mission or an operation stage. Each service gets its own repeat-protection key derived from the operation.

Can a rule hold a run until a person says yes?

Yes. A rule that needs a person (for example, "routing over $500 needs a person") does not refuse the run. Router saves the request exactly as it would be sent, sends nothing, and answers 202 with status: "waiting_for_a_person", the decision to open (decision_id, decision_path) and when the wait ends (expires_at: the rule's own limit, else seven days).

202
{ "status": "waiting_for_a_person", "decision_id": "governance_task:…", "decision_path": "/infra/dashboard/decisions/governance_task:…", "expires_at": "…" }

The person opens it in Decisions and sees what will be sent: the job, the service, the amount and each plain value of the request. Only a person can decide it (an API key gets person_required). If they approve, the run goes once, sending only the request they were shown. If they decline, or nobody answers in time, nothing is sent.

Sending the same request with the same Idempotency-Key again says where it stands: still 202 while it waits; after it started, 409 run_already_started naming the operation to read; if it was declined or ran out, 409 held_run_ended with a sentence, and a new run needs a new key. A rule that refuses (not holds) is still 403. The run's receipt says who approved it.

You don't need a model to write the common rule. In the dashboard, a job's "What limits apply?" takes an amount, a currency, optionally one service, and what happens. From a script, with a key that holds policies.write:

POST /api/v1/cadreen/routing/limits
{ "amount_over": 500, "currency": "USD", "service": "Wise", "effect": "needs_a_person" }

effect is needs_a_person: the run waits in Decisions until a person says yes. A limit also counts as a rule that covers routing, so a workspace that had none no longer refuses every run, and this limit won't ask for runs under it. Other rules, and a job in another currency, can still hold a run. It is in one currency: a job in another currency is held for a person when it could be over. The limit is one of the workspace's own rules; see it and change it under Policies.

What comes back?

The run: status, the service that did the work (selected_provider), one sentence in explanation with a stable explanation_code, and result. In result, raw is the service's answer as it came, and normalized is the answer in the job's own result shape when the job has one, the same whichever service answered. Build on normalized.

How can one job use services that want different requests?

Services rarely agree. One wants the amount in cents as value_minor, another wants it in dollars as amount, one answers COMPLETED and another succeeded. A job can say what it takes and gives back: its shape. Each service then says how it speaks that shape. The person calling the job sends one request and gets one kind of answer, whichever service did the work, and adding, removing or swapping a service changes nothing for them.

A job with no shape works as it always did: the request goes to each service as it is sent, and the service's answer comes back as it is.

How do I say what a job takes and gives back?

In the dashboard, open the job and choose "Say what it takes and gives back". Or send it:

PUT /api/v1/cadreen/routing/groups/{id}/shape
{ "shape": {
    "input":  { "amount":   { "type": "amount",   "required": true },
               "currency": { "type": "currency", "required": true },
               "account":  { "type": "string",   "required": true } },
    "result": { "status":    { "type": "status" },
               "reference": { "type": "string" } } } }

A field is a string, a number, a boolean, an amount, a currency or a recipient (who receives the work, such as an account, as text). A result may also have one status: the word Router reads to know whether the work succeeded, failed or is still pending. Mark a result field a service may not give as optional. A shape has up to 50 input and 50 result fields, and a name is letters, digits, _ and -, up to 64.

From then on every request is checked before any service is looked at. A missing field, a field the job doesn't take, a wrong type or a currency that isn't real is 400 routing_input_invalid, naming the field, and nothing is sent.

A person sets the shape: a workspace admin or the owner, signed in. An API key gets 403 person_required, because the shape decides what reaches every service.

How does a service say how it speaks?

In the dashboard, open the job and choose "Set up this service". One page holds how the service speaks, what is true of it and how it stands in, with one Save that asks you once. It lists what the service says it takes, from its own connection however it was connected, and puts what it needs first. It never fills anything in for you: a field with the same name as one of the job's is only suggested until you choose it. Or send it:

PUT /api/v1/cadreen/routing/groups/{id}/providers/{provider_id}/mapping
{ "mapping": {
    "input":  [ { "to": "value_minor",  "from": "amount",   "transform": "amount_minor" },
               { "to": "ccy",          "from": "currency", "transform": "lower" },
               { "to": "account_ref",  "from": "account" } ],
    "result": { "reference": { "path": "transfer.ref" },
               "status": { "path": "transfer.state",
                           "values": { "COMPLETED": "succeeded", "REJECTED": "failed", "PROCESSING": "pending" } } } } }

Each entry of input says where something goes in the service's own request (to) and what goes there: a field of the job's request (from) or a fixed value (const), changed if you ask (upper, lower, string, amount_minor or amount_major). Each field of the job's result says where to read it in the service's answer (path). For the status, say what each of the service's own words means: succeeded, failed or pending. A word you leave out is never guessed: the run stops for a person to check.

Cadreen checks what you wrote against the service's own description of what it takes when you save it, and again before every send. A mapping has up to 50 entries on each side and 20 status words, paths up to 6 parts deep, a fixed value whose JSON text is up to 1,024 bytes (its quotes, and the backslash before each quote or backslash in it, count), and 16 KiB in all. The fixed numbers together may be at most 4,096 digits written out in full (1e5 is 6), and text the database can't store (a null character, half of an emoji) is refused: both are 400 invalid_mapping, with a sentence that says which.

Credentials never go in a mapping: a fixed value is shown to everyone who can read the job. A property the service marks as a secret (a password, an auth token, headers) is refused, and so is one Router adds itself, such as idempotency_key. Set a secret in the service's connection. A person sets a mapping too, as for the shape.

What about amounts?

An amount is a number in the job's own unit, so the job has to say whether it counts in minor units (cents) or major units (dollars):

PUT /api/v1/cadreen/routing/groups/{id}/amount-unit
{ "amount_unit": "minor" }

Router converts with the currency's own number of decimals and never rounds. A service that would need an amount rounded isn't sent the request: it is skipped, and the next is considered. A service whose mapping converts amounts, in either direction, can't run on a job with no unit: it is skipped before anything is sent, and the job says so. A person sets the unit, as for the shape.

What happens when something doesn't fit?

• A service nobody has said how it speaks is skipped (mapping_missing). One whose request can't be made in its shape is skipped too (mapping_failed), and mapping_problem says which field and why. Router considers the next service. If none can be used, nothing was sent (nothing_tried).
• On a job with a recipient field, a service whose mapping doesn't send that field is never sent the work, on any run (recipient_not_taken): it would pay some other account, or none. If no service takes the recipient, the run stops before anything is sent and says so.
• A request missing something a service's tool requires is never sent (missing_parameters). The run stops and names what is missing. On a job with no shape another service would be sent the same request; on a job with a shape, the request its mapping made lacks what this service's tool requires, which is the mapping's to fix, not another service's to try.
• A request with something a service's own spec gives no way to send is never sent to it (request_unsendable): for example a list where the spec takes one value, or a body in a format Cadreen doesn't send yet. Router moves the work to an equivalent service, because that service has its own spec, and there is nothing to check, because nothing left Cadreen.
• A value a service's tool doesn't take as it is, text where it takes a number or a value that isn't one of its choices, is never sent either (arguments_refused; refusal_code says which). The run stops, and there is nothing to check. On a job with no shape another service would be sent the same value; on a job with a shape, the value its mapping made doesn't fit this service's tool, which is the mapping's to fix, not another service's to try.
• A tool built into Cadreen that reads numbers as decimals can't hold a whole number above 9,007,199,254,740,992 exactly (builtin_cant_read_number). Nothing is run. Send it as text, or use a connected service's tool, which takes the number as written.
• If a service answers and Cadreen can't read the answer (a status word the table doesn't hold, a field that isn't there), the run stops and says a check is needed: the service may have done the work, and no other service is asked.
• A service whose status says failed stops the run, and Router asks no other service on its word. A pending one makes the run wait for the service.

The job says what is wrong in plain sentences, so you don't have to wait for a failed run. The same read also says how many real runs the job has had in your workspace (real_runs; null when it couldn't be counted), the small amount the dashboard offers for a first real run (first_run_amount, in the job's amount unit, or null), and how many decimals the job's currency has (currency_exponent):

GET /api/v1/cadreen/routing/groups/{id}

shape_warnings is an empty list when nothing is wrong. Otherwise it holds provider_unmapped, mapping_invalid (a saved mapping that no longer fits the shape), amount_unit_undeclared, no_mapped_provider (no service can take the job's requests, so it sends nothing), recipient_not_taken (a service isn't sent the account a run names, so Router never sends it work), operation_changed (the API's description of a service's step changed since the job was set up) or shape_unusable, each with a sentence and, for a service, its name. The dashboard shows them on the job and on each service.

A job with no shape sends each service the request as it is. If a service's spec doesn't take a field your runs send, that service is skipped and the work goes to another one, or the run stops. The job then says so: provider_field_not_taken names the service and the fields (fields, never their values), from the runs of the last 7 days, at most the 200 most recent. A preview (dry-run) checks the same thing before anything is sent and names the fields on the skipped service (unsendable_fields). Leave those fields out, or give the job a shape so each service gets only its own fields.

To see what a service would be sent, preview the job: POST /api/v1/cadreen/routing/groups/{id}/dry-run chooses a service and shows the request in that service's own shape, and calls no service. With no usable service it is 409 no_mapped_provider.

When does Router try another service?

Only when both are true: the first service failed before it accepted anything (it refused, was rate limited, rejected the sign-in or provably couldn't be reached), and you have set up the other service as a backup for it. A timeout, a 5xx (a 503 too) or a failure Cadreen can't read stops the run instead, so Router never sends the work on again on a guess.

A 503 is very often sent by a gateway, a load balancer or a CDN in front of the service, after your request already reached it, so the payout may have been made. Router treats it like a 504: the run stops and a person checks the service (reconcile_required). A service whose own documentation says its 503 means the request was not processed can say so, and then a 503 moves the work as before. A workspace admin, signed in, declares it on the service's failure map; an API key can't, and every change is audited. The link is the page of the service's documentation that says so. It covers 503 only: a 502, a 504 and a timeout still stop.

PUT /api/v1/cadreen/routing/groups/{id}/providers/{provider_id}/failure-map
{ "failure_map": { "unavailable_means_not_processed": true,
  "unavailable_doc_url": "https://docs.example.com/errors#503" } }

A service you connect from an OpenAPI description is read the same way: its refusals (a 403, a 404, a 410) are refused before anything was accepted and may move the work to a backup; a 409, a 500, a 502, a 503 and a 504 stop it, and a 422 says the request itself isn't valid.

Say a service is a backup for another by giving both the same side_effect_class. For work with side effects, also give them the same equivalence_class and an idempotency_support other than none, and someone has to stand behind that claim for each of them: either the service's tool confirms how it passes Cadreen's repeat-protection key on, or a workspace admin, signed in, accepts the risk that it doesn't with accept_idempotency_risk. Without one of the two, Router doesn't move work between them. Services start as unknown, which never fails over.

A service can't be declared read_only when its own connection says calling it changes things: the save is refused with 409 side_effect_contradicts_tool. Declaring it as changing things is always allowed, and where the connection says nothing your declaration is accepted and stays as you saved it.

Router treats a service whose connection says nothing as if calling it changes things, whatever you declared. It moves work only the way a service that changes things does: repeat protection on both services, one equivalence_class, and someone standing behind each claim. A read-only service whose connection says nothing is not a backup for one whose connection says it only reads. A service you declared read-only whose connection says it changes things, says nothing and has no repeat protection, or can't be read, is passed over as not_equivalent, and the attempt's class_problem names the service and what its connection says: tool_says is effectful, undeclared or unreadable. When that is why a run stopped, its own sentence says so (explanation_code stopped_class_problem), with what to do: declare the service as changing things and set up repeat protection, or have its connection say it only reads.

A service set up from a template Cadreen ships may take its repeat-protection key in a header the template names, such as PayPal-Request-Id. Its record then has idempotency_header, idempotency_key_format (opaque, or uuid for a service that only takes one) and idempotency_declared_by (the template and its digest); otherwise all three are null. Such a claim is backed, and Router sends the key in that header.

You can see it before a run does. Each service you read says what its own connection declares (tool_side_effect_class: read_only, effectful, unknown when the connection says nothing, or null when Router couldn't read it) and carries the same class_problem. The dashboard shows it on the service, and its backup preview never says Router may switch to a service that Router would refuse.

POST /api/v1/cadreen/routing/groups/ensure takes only name, capability, description, auto_discover and providers (each with tool_id, name, connector_type and priority). A field it doesn't read, such as a service's side_effect_class, is refused with 400 unknown_field naming it, and nothing is created. Declare how a service stands in with the PATCH below, after the service is added.

PATCH /api/v1/cadreen/routing/groups/{id}/providers/{provider_id}
{ "side_effect_class": "effectful", "equivalence_class": "usd-payout", "idempotency_support": "header", "accept_idempotency_risk": true }

Can Router choose the cheapest provider?

Yes, when you choose price. By default Router ranks on health, success rate, speed, priority and its catalog's rating, and price doesn't count. A workspace admin gives cost a weight:

PUT /api/v1/cadreen/routing/groups/{id}/scoring
{ "health": 0.2, "success_rate": 0.2, "latency": 0.1, "priority": 0.1, "catalog": 0, "cost": 0.4 }

A provider's price for a request is what its declared fees come to: fee_fixed plus the request's amount times fee_percent. You declare the fees on each provider (PUT .../providers/{provider_id}/facts) and the job's currency, and the amount is the field the job's shape declares as an amount (a job with no shape is priced on fee_fixed alone). The cheapest eligible provider scores 1 for price and the dearest 0. The sum is exact, and Router never converts one currency into another.

Does a provider charge in another currency? Say so in its facts with fee_currency (for example "EUR"). Without it, its fees are taken to be in the job's currency. Router compares prices only in the request's currency, so a provider whose fees are in another one isn't compared on price: it doesn't count as the dearest, it is ranked on everything else, and the record of the choice says why ("fee in EUR, job in USD"). Its runs record no cost. To have price count for it too, declare its fees in the job's currency. A request in another currency than the job's is compared on price only among the providers whose fees are in that currency; the others aren't compared.

A provider that hasn't said what it charges scores 0 for price, so it can't win on price by saying nothing, and the record of the choice says so ("Adyen hasn't said what it charges"). With cost at 0, which is how every job starts, nothing changes and the record has no price in it. Changing the weights is recorded as a change to the job, with who made it.

How do I see what a run did?

In the dashboard, "What happened recently?" below the jobs lists the runs. Or list them:

GET /api/v1/cadreen/routing/executions

Each run has one plain sentence in explanation and one record per service in attempted_providers, in the order Cadreen considered them. A service with outcome: "skipped" was never sent the request: it has not failed, and skip_reason_code says why. A failed attempt says why in category, and status_code is the HTTP status the service answered with. When no status came back, transport says what the connection did: reset (it was cut), closed (the service closed it) or deadline (nothing came back in time). A run that was a test has test_run: true, and GET /api/v1/cadreen/routing/numbers counts test runs apart (test_runs) and leaves them out of every other number. Add group_id to /routing/numbers or /routing/safety for one job's runs (a job you can't see is 404 group_not_found). Each service's numbers include failovers_to: the runs it did after the work moved to it. When failure.reconcile_required is true, the service may already have done the work: check it before trying again.

Runs are listed for your workspace only. Use linked_operation_id to see the ones that belong to one operation.

How do I hear about what Router did?

Subscribe a webhook. Router sends one event when a run ends or waits: router.run.succeeded, router.run.failed (also when a run was refused before anything was sent: nothing_sent is true), router.run.needs_check (nobody knows whether the service did the work, so a person must check) and router.run.waiting (the run waits for the service's result). When a service declares how to look up a request, Cadreen checks for you: a run it settles that way sends router.run.succeeded with decided_by set to lookup, and if it stops looking it sends router.run.needs_person with a cause. Each of the four carries test_run: true when the run was a test sent only to test services, so a webhook that acts on a success can tell a test from real work. It also sends router.failover when work moved to a backup, router.provider_unavailable when a service starts resting, and router.rule_invalid when a saved rule no longer reads.

POST /api/v1/cadreen/webhooks
{ "url": "https://example.com/hooks", "events": ["router.run.succeeded", "router.run.needs_check"] }

Every event is signed and carries an event_id. One can arrive twice: drop an event_id you have seen. The signing secret is shown once, when you make the webhook. Each delivery is listed at GET /api/v1/cadreen/webhooks/{id}/deliveries, and GET /api/v1/cadreen/routing/events lists every Router event and when it was handed to the webhooks.

Can I set it all up from a script?

Yes, with one key and no person, as long as the key holds the scopes for what the script does. The exception is what decides what reaches every service: a job's shape, how each service speaks, the amount unit and the currency are set by a person, an admin or the owner, signed in, and a key gets 403 person_required. An admin gives a key its scopes when creating it:

• routing.setup creates jobs and changes their services, backups and rules.
• connections.write connects an API or a tool server to the key's own workspace, with a credential of that workspace. It can't use the organization's own key. Together with a write level it lets the key connect a service to any host it names, with any credential of its workspace: a credential that names its host (allowed_host) is only sent there.
• policies.write writes rules, and chooses how routing starts (POST /routing/starter-rules).

A key can declare a backup whose repeat protection the service's own connection confirms. It can't make a claim Cadreen can't check, or accept that claim's risk: a person does that. Every change records who made it, the person or api_key:<id>. A key without a scope gets 403 scope_required naming it.