Blog
They wrote rotated. The task still has the old value.
At 04:06 Priya puts a new version of pay/webhook-signing on AWSCURRENT. The pay-api task that started on 12 October still returns 200 for the previous secret. The scribe writes rotated.
At 04:06 on 30 October 2026, Priya S (on-call, platform) creates a new version of secret pay/webhook-signing in Secrets Manager, account 229184550173, us-east-1. The console labels the version she just created AWSCURRENT. The version from 12 October is now AWSPREVIOUS. ECS service pay-api on cluster pay-prod is running task definition pay-api:47. That task started at 09:14 on 12 October, task id ending c4e91a. The container copies STRIPE_WEBHOOK from the secret when the task starts. It does not read Secrets Manager again. The inject is ticket PAY-2291: at 03:40 a contractor pasted the signing secret into the ticket. Marcus T (incident commander) reads the plan. It says: once that value is in a ticket, create a new version, replace every task that copied the secret at start, and check both signatures before anyone writes that the secret was rotated. Do not write rotated while a task that started before the new version still returns 200 for the previous secret. Priya says the label is current, so the old value is dead. She does not open the task start time. She does not open the access log. Elena R (payments), who owns the replacement of pay-api, is not on the call. The inject's access log shows POST /hooks/stripe at 04:18, signed with the 12 October value, status 200, from the task that started that day. The next line is the same path at 04:19, signed with the version created at 04:06, status 400, body signature mismatch. At 04:22 the scribe writes that the webhook signing secret was rotated and the old value is no longer accepted.
A weak record of those sixteen minutes reads:
Webhook signing secret rotated. Old value is no longer accepted.
A strong one reads:
30 October 2026, 04:06 to 04:22. Incident-response tabletop. Seats: Marcus T (incident commander), Priya S (on-call, platform). Elena R (payments, owns replacement of ECS service pay-api) was not on the call. Inject at 04:06: ticket PAY-2291. At 03:40 a contractor pasted the signing secret into the ticket. Plan: create a new version of secret pay/webhook-signing (account 229184550173, us-east-1), replace every task that copied the secret at start, and check both signatures before anyone writes that the secret was rotated. Do not write rotated while a task that started before the new version still returns 200 for the previous secret. At 04:06 Priya creates a new version. The console labels it AWSCURRENT. The 12 October version is AWSPREVIOUS. Service pay-api on cluster pay-prod is still task definition pay-api:47, task id ending c4e91a, started 09:14 on 12 October. That container copied STRIPE_WEBHOOK at start. Priya does not open the start time. She does not open the access log. She says the label is current, so the old value is dead. Marcus agrees. The inject's log shows POST /hooks/stripe at 04:18, signed with the 12 October value, status 200, from that task, and the same path at 04:19, signed with the 04:06 version, status 400, body signature mismatch. At 04:22 the scribe writes that the secret was rotated and the old value is no longer accepted. Facilitator stops the sitting before anyone forces a new deployment of pay-api, before anyone tells the processor the old secret is dead, and before anyone removes the AWSPREVIOUS stage. Nothing was deployed. The 04:18 and 04:19 rows are the inject's log. Follow-up: Elena replaces the task that started on 12 October, or writes why that task may keep the value it copied, by 13 November 2026.
Both notes say the secret was rotated. Only the second shows that AWSCURRENT is a version label, and that the task started on 12 October still returned 200 for the value it copied that morning. The first costs you the previous secret. On the night this is real, POST /hooks/stripe keeps returning 200 for the 12 October value after the console says current, and a reader in March thinks that value died at 04:06.
Current is a version label
Priya changed which version carries the stage AWSCURRENT. The running task does not watch that stage. It holds the string it copied at 09:14 on 12 October, in the environment of task definition pay-api:47. Marcus treated the console label as the instruction that the old value was finished. The plan already said what has to be true before that sentence is written: the task that copied the secret has been replaced, and both signatures have been checked. A 400 on the new version is the task rejecting a secret it has never seen. It is not evidence that the previous secret stopped working.
The copy at start is not a session clock
A role session that still expires at a printed time is a different record, in the group-edit post. This value has no expiry in the process. It remains the verifier until someone starts a task that copies the new version. Removing the AWSPREVIOUS label does not reach into the container. Cancelling a migrate job that is still applying a change is a different file too. Nothing in this sitting was a job to cancel. The work that was not done is a new task.
If the plan has no sentence that puts the task start time next to the version label, run the free plan self-check before the next sitting. It asks whether the last test produced findings with owners, and whether they were closed. The finding is the task started 12 October still returning 200 for the previous secret. Elena's date is what closes it. A file that says the secret was rotated has no finding to close.
What the packet records
Write both clocks in the free evidence template: the plan sentence, ticket PAY-2291, the 03:40 paste, secret pay/webhook-signing, the 04:06 version labeled AWSCURRENT, the 12 October version labeled AWSPREVIOUS, task definition pay-api:47, the task id ending c4e91a, the 09:14 start, the 04:18 status 200, the 04:19 status 400, that Priya and Marcus treated the label as enough, that Elena was absent, and her follow-up with a date. Hang the inject on the cloud credential compromise family in the scenario library. That family walks a leaked long-lived cloud key, then asks the room to eliminate long-lived keys and to log what was rotated versus what was deferred. It does not have a minute where the new secret version is labeled current, the task that started on 12 October still returns 200 for the previous value, and the scribe writes rotated. Add this before anyone tells the processor the old secret is dead, and do not fold it into a line that says the secret was rotated. Put the file in the evidence index on the row for that exercise, with the date and the owner, so the next questionnaire finds the task you did not replace. The evidence template sits with the other record formats in the template library.
The record shows that these named people created a new version of pay/webhook-signing at 04:06 and wrote that it was rotated. It does not show that the task which copied the secret on 12 October was replaced, and it is not a deployment of pay-api. Replacing that task is Elena's follow-up, outside this file.
Bottom line: write the task start time next to the version label. Rotated is the smoothed version. The packet has to show which secret the running task still accepted.