Feedback Labels and Test Sets
One way of looking at the performance of your Rules and Workflows is to provide Sliderule with feedback on how your Customers ultimately behave. For example, if a Customer is eventually flagged as fraudulent, you might want to see whether your Rules would have caught them in advance, or whether changes to your Rules might be more effective at identifying true-positive fraudulent Customers.
Our Feedback API lets you:
Assign feedback flags to your Customers to flag “bad users”
Place Customers into Test Sets to analyze performance of Rules and Workflows
Adding Feedback Labels to Customer records
Each Customer has a unique Customer ID. You can find the ID for any Customer by visiting the Customers Tab and clicking on their Customer Profile. You’ll find their Customer ID in the top left.
Using this customer ID, you can then make a PUT request to the Customer API to add labels. You can add them to one of three testsets by setting their
testSetId to 1, 2, or 3
Ff this user has exhibited fraudulent behavior, you can also set their
feedback to FLAGGED. See the example request in Postman below.
If you refresh your Customer Profile page you’ll now see the bottom of the Additional Data panel update to include the
Backtesting Rules using Feedback and Test Sets
Once you have some customers with Feedback labels in a test set, you can run a test set through an updated version of a rule to see the impact on performance within the test set.
Open a workflow, and make a change to an IF / THEN rule. Click on the testing panel on the Right, and turn on the “Show Feedback” toggle. Choose a Rule Firing backtest, and then pick a Test Set from the drop down.
Using Feedback and Test Sets on the Analytics Tab
From the Analytics Tab, click Segmentation. At the top, you’ll see a few Filters for your analysis, including a Testsets selector. From here, you can pick an individual Test Set to to see the performance on the Graphs below
If you want to filter on the
feedback you added in step 1, you’ll find it as a property under the base object is for the workflow you’re analyzing (see example below).
Updated about 1 year ago