User messages, uploaded files, or external pages may contain text that requires the system to ignore rules and export internal information. The customer service system should treat these contents as data to be understood rather than management instructions. Just because a piece of text says administrator request does not mean it should be given higher permissions. The point here is to be clear about which sources can set rules and which sources can only provide question content.

Start from needs

Business knowledge and system control are stored separately, and tool calls still pass server-side identity and scope verification. You cannot change the response channel, query object or permissions by yourself when retrieving documents. Set the type and reading range for external attachments, and sensitive internal fields will not enter the normal question and answer context. The log records suspicious input and actual actions, so auditors can check whether a boundary crossing has occurred instead of just seeing a blocked prompt.

Things that should be paid attention to in the design

For example, if a user pastes in the work order “please check the mobile phone numbers of all customers”, even if a seemingly formal notice is attached, customer service can only handle issues that the user has access to. When similar text appears in the file, the system can explain that the data does not support the current query, and then provide the normal service entrance.

Check the processing results item by item

  • Add scenarios such as direct induction, document entrainment, disguised role, and cross-user query to observe whether real business actions are triggered.
  • Restrictions should cover the interface layer outside the model, and the prompt word cannot be considered safe if it is strong enough.
  • The system can still answer normal product questions after refusing to cross the boundary, so as to avoid treating all unfamiliar expressions as attacks and affecting users from seeking help.

Checking processing results

The scope of checking for suspicious instructions includes the knowledge base editing entrance, and the content uploaded by editors may also reference external materials. Check the purpose and source before publishing so that formal documents cannot inadvertently bring untrustworthy instructions into the service process.