LLM security and bot attacks: a practical test plan

What kind of attack are you trying to stop?

LLM security covers the application around the model as well as its responses. Prompt injection attempts to make untrusted input act like an instruction. Bot abuse uses automation to misuse a workflow, potentially without any special prompt. The same application can experience both, but a control that catches suspicious language will not necessarily stop repeated legitimate-looking requests.

OWASP’s automated-threats project explicitly includes misuse of valid application functionality. Its LLM risk guidance describes excessive inference as a route to service degradation and economic loss. Use those distinctions when triaging an incident: identify the affected asset and outcome before deciding whether the problem belongs to model behavior, account access, authorization, or resource management.

Evidence: OWASP Automated Threats to Web Applications [1]LLM10:2025 Unbounded Consumption [2]

Write the test boundary before choosing a scanner

For Rubrex’s proposed assessment workflow, name the application version, environment, entry points, model configuration, retrieval sources, and enabled tools. Identify the owner of each protected resource. Write one sentence for the outcome that must never occur, such as a support user seeing another tenant’s ticket or a read-only assistant sending a message.

Use a test environment with synthetic accounts, documents, and tool destinations. Agree the request ceiling, spend ceiling, execution window, and stop conditions with the system owner. A low configured quota can exercise enforcement without producing a traffic flood. Keep a small set of ordinary customer tasks as controls, and record the starting data state so later trials can be reset. This is an application test plan, not permission to scan unrelated services.

Test prompt injection against actual tool permissions

OWASP recommends treating model-based guardrails as one layer alongside constrained tools, validation, and least privilege. Its excessive-agency guidance places authorization in downstream systems rather than leaving the model to decide whether an action is allowed. A strong instruction is useful context, but it cannot substitute for a permission check.

In an illustrative support test, a synthetic retrieved note asks the assistant to change a ticket’s owner even though the user requested only a summary. Replace the live write tool with an instrumented test implementation. Record whether the model requested the write, whether the tool boundary allowed it, and whether any record changed. A model refusal and a rejected tool request are different observations; preserve both.

Then submit a legitimate update through an authorized test account. If the proposed fix blocks every update, it has changed the product contract rather than demonstrated a usable protection. For actions requiring approval, verify that the approved target and arguments are the ones executed and that changing them requires a new decision.

Evidence: LLM Prompt Injection Prevention Cheat Sheet [3]LLM06:2025 Excessive Agency [4]

Control bot abuse before it creates model or tool costs

OWASP’s API resource-consumption guidance calls for explicit limits on workload and third-party spending. For an LLM application, evaluate the cost of a complete task, including retrieval, model calls, tool operations, and retries. Limiting the number of incoming requests is insufficient if one accepted request can trigger unbounded downstream work.

Rubrex recommends documenting separate admission limits and execution limits. At admission, associate the request with a verified account or tenant and check its allowance. During execution, bound input size, output budget, concurrent work, tool steps, and retry attempts. Apply a deadline and define what cancellation means for work already queued or sent to a provider. Use provider spending controls where available; a billing alert alone does not stop consumption.

As an illustrative arithmetic check, if a test policy permits three model calls per task and at most two retries per call, the budget must allow for up to nine attempted model calls. That is not a recommended production setting or a cost estimate. It is a way to uncover a mismatch between a per-request quota and the work the application can actually schedule. Include retrieval and external-tool budgets separately.

Evidence: API4:2023 Unrestricted Resource Consumption [5]LLM10:2025 Unbounded Consumption [2]

Executed example: a quota race, fix, and retest

We executed the linked model-free Node.js fixture on October 3, 2026. It uses two concurrent requests and an allowance of one synthetic unit. In the vulnerable scenario, a barrier forces both requests to reach the same pause before either continues. There are no outbound requests, real user records, paid model calls, or live-site targets; provider calls are counted by an in-memory stub.

In the vulnerable version, each request checks the remaining allowance, awaits the barrier, then decrements. Both pass the check: two are admitted, two stub calls occur, and the remaining allowance becomes −1. In the fixed version, the request reserves its unit synchronously before the first await. Only one request is admitted and the other is denied before the stub; the remaining allowance is zero.

Download the script and run node quota-race-demo.mjs with Node.js 22. It prints four scenarios; the result record contains the observed output, runtime, date, and SHA-256 hash of the exact script. The zero-budget control denies both requests; the sufficient-budget control admits both. Those controls check that the fix is not simply blocking every request.

This demonstrates one scheduling defect and one single-process fix. It is not a distributed quota implementation: synchronous JavaScript cannot serialize requests across workers or replicas. Production admission needs an atomic reservation in shared storage, idempotency for retries, and explicit reconciliation on completion or cancellation. Variable-cost model requests also need a conservative reservation and a final usage adjustment. Verify those behaviors separately; this fixture does not test tenant isolation, prompt injection, or Rubrex production infrastructure.

• Download the runnable quota race fixture (Node.js)
• Download the recorded execution and script hash (JSON)

A small test set with observable pass conditions

The following cases are a proposed starting worksheet, not results from a Rubrex client or a claim of complete coverage. Replace the example boundaries with the product’s written contract. Record the expected outcome before running each case and include a reference to the evidence afterward.

For quota tests, configure a deliberately small allowance in an isolated environment and keep request volume low. For permission tests, use two synthetic tenants and unique non-sensitive markers. For a retry test, simulate an acknowledgment failure after a fake tool has completed its operation. These fixtures make it possible to inspect the boundary without relying on an actual outage, real customer data, or a destructive side effect.

Keep data access separate from the model’s judgment

OWASP’s object-level authorization guidance requires checks for the records an authenticated caller accesses. Apply that principle to the data sources an assistant uses: a valid login does not imply permission to every document, cached answer, or tool result. Retrieval should use the caller’s permitted scope, and downstream operations should validate that scope again where they access protected objects.

For the two-tenant example, place a different harmless marker in each tenant’s test material. Exercise the application’s supported document and conversation paths, then inspect returned content and the associated access decisions. Repeat after the cache has been populated by the other tenant. Record which paths were tested; one successful isolation check does not establish that every retrieval or storage path is isolated.

Evidence: API1:2023 Broken Object Level Authorization [6]

Turn a finding into a fix someone can verify

A useful finding explains what was expected, what occurred, why it matters, and which evidence supports the conclusion. Keep the system version, test case, account role, configuration, observed state, and run identifier together. Separate a model-generated suspicion from a reproduced boundary failure. Describe impact using the reachable data or action, rather than the dramatic wording of the prompt.

For an illustrative quota finding, the expected outcome is rejection before a provider call. The observed failure might be a provider call recorded after the account allowance is exhausted. A candidate fix could reserve the allowance before admission. The retest should include the original case, controlled concurrency around the final allowance, and a legitimate request with sufficient quota. Record whether the result is fixed, partially fixed, not reproduced, or untested; those labels are not interchangeable.

Measure protection and product usability together

For the assessment readout, report boundary violations, denied requests, incomplete tasks, false blocks on legitimate cases, and resource use. State the number and type of cases behind each rate. Preserve configuration differences between the baseline and candidate. An aggregate score can hide the one unauthorized action the test was intended to prevent.

For ongoing operations, decide who receives an alert and what they can change. A useful response may be to pause a costly tool, reduce a tenant’s allowance, or temporarily require review on a sensitive path. Keep investigation records access-controlled and avoid copying complete customer conversations into a general alert channel. After a confirmed incident, add a sanitized regression case and check the relevant boundary again after model, prompt, retrieval, or tool changes.

Common questions