TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
ServiceNow CoreAI describes AutoSynthData, a pipeline that uses a target agent’s failures and a stronger teacher’s successful runs to generate new enterprise tasks for training. The report says tasks are grounded in an environment and checked for feasibility and solution validity; it does not provide quantitative evidence of improved performance in the supplied material.
ServiceNow CoreAI has described AutoSynthData, a pipeline designed to generate training tasks for enterprise agents from the weaknesses a target model shows in a specific environment. In a report published on Hugging Face, the team says the system uses a stronger teacher model to help identify solvable gaps, creates new tasks that exercise those capabilities, and checks the tasks in the environment before using accepted examples for post-training.
The report frames the problem as a mismatch between broad model ability and the demands of a particular workplace system. An agent may have access to tools and data yet fail at a specific workflow, misuse a combination of tools, or disregard a policy constraint. A single failure can reveal a weakness, but the team says training calls for many varied tasks that test the same capability while remaining realistic and possible to complete.
AutoSynthData represents each task as a system specification, user prompt, and verifier. The system specification sets the operating rules and relevant environment state; the prompt describes the user’s request; and the verifier checks whether the final result meets the request without violating constraints. The report stresses that tasks should be feasible and realistic, while still challenging the current model. Verifiers should reject failures but accept valid solutions rather than insist on one prescribed sequence of actions.
For its example, ServiceNow says it used EnterpriseOps Gym, an enterprise-agent environment, and a released dataset. The pipeline evaluates a target model and a stronger teacher on diagnostic tasks, summarizes the findings in “sanitized capability specification cards,” then generates different tasks from those cards. According to the report, the generator is not given the original evaluation prompts, entities, trajectories, or verifier details. Generated tasks are checked in the environment, and accepted samples can be used in post-training. The updated model can then be evaluated to identify remaining gaps.
Training Against Workplace Failures
The approach addresses a practical challenge for organizations deploying agents: general capability does not guarantee reliable performance in a company’s particular systems, policies, and data. A training task that ignores those conditions may not prepare an agent for the work it will actually encounter.
AutoSynthData’s proposed loop links evaluation to task creation. If it works as described, it could let teams focus training on workflows an agent currently handles inconsistently, while varying the prompts and environment details to avoid simply repeating an evaluation example. That matters because an agent’s output can change system state, and a weak training signal can reward an incorrect action or penalize a valid one. The report accordingly treats verifier quality as a core part of the method, not just a scoring detail.
The report describes a pipeline and an illustrative experiment, rather than establishing that the method improves performance across enterprise deployments. Its practical value will depend on whether generated tasks reflect real work and whether the checks reliably distinguish successful outcomes from mistakes.
enterprise AI training data generation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How AutoSynthData Builds Tasks
The report starts from the environment in which an agent operates: the state it can observe or change, the tools and APIs available to it, and the effects of its actions. A useful task must fit those capabilities and constraints. For example, a request cannot be a valid training example if it requires an unavailable tool or a state change the environment does not support.
ServiceNow’s process separates discovering a weakness from generating training material. Diagnostic runs show where the target model fails and how the teacher succeeds; the team then records the relevant capability, workflow, and success conditions in a capability card. The generator uses that description to produce new prompts and task states. The report says the original evaluation examples are withheld from generation, a design intended to create fresh tasks rather than reproduce the diagnostic set.
After generation, tasks are run and checked in the environment. This step is meant to screen out examples that cannot be completed or whose verification does not match the stated request and system rules. The report describes repeating evaluation after post-training so the next generation round can focus on the weaknesses that remain. The supplied material does not specify a release schedule or deployment beyond the EnterpriseOps Gym example.
“A model may be broadly capable and still struggle with a particular environment.”
— ServiceNow CoreAI, in the AutoSynthData report
automated training data creation for machine learning
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Evidence and Deployment Details
The supplied report material does not include quantitative results showing how much AutoSynthData improved the target model, how it compared with other training approaches, or whether gains held across different enterprise environments. It also does not provide details here on the scale of generated data, the cost of generation and checking, or how often human review is required.
ServiceNow describes the capability cards as sanitized, but the supplied text does not explain the specific safeguards used to prevent sensitive enterprise information from entering generated tasks. The report also does not establish that the method is deployed broadly in customer systems. Those points remain open; the described work should be read as a report on the method and its EnterpriseOps Gym example, not as proof of production impact.
enterprise agent environment simulation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Further Evaluation of Generated Tasks
The next step described by the team is iterative evaluation: post-train on tasks accepted by the environment checks, run the updated model again, and use its remaining failures to guide another generation round. That process would test whether the curriculum shifts as the model’s weaknesses change.
To judge the method’s broader value, readers would need results that report task-generation and verification quality alongside model performance, as well as comparisons across environments and training baselines. The supplied report excerpt does not give a date for such results or identify a forthcoming release. Until further evidence is available, the demonstrated scope remains the stated EnterpriseOps Gym example.
AI model testing and validation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is AutoSynthData?
AutoSynthData is a training-data pipeline described by ServiceNow CoreAI. It uses a target agent’s failures and a stronger teacher’s successful behavior to generate new enterprise tasks, then checks those tasks in the environment.
How does the system decide what tasks to generate?
It evaluates the target model on diagnostic tasks, identifies capability gaps, and records them in capability specification cards. The generator uses those cards to create different tasks, rather than receiving the original prompts and trajectories, according to the report.
What makes a generated task suitable for training?
The report says a task should be feasible, realistic, and difficult for the current model. It also needs a verifier that checks whether the outcome meets the request and constraints without requiring one exact solution path.
Does the report show that AutoSynthData improves agent performance?
The supplied material describes the method and its use with EnterpriseOps Gym, but does not provide quantitative performance results. The size and generality of any improvement are therefore unclear from this report excerpt.
What happens after the tasks are generated?
Generated tasks are checked in the environment, and accepted examples can be used for post-training. The updated model is then evaluated so the team can identify which capability gaps remain and guide another round of task generation.
Source: rss
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
