Feedback Labels and Test Sets

One way of looking at the performance of your Rules and Workflows is to provide Sliderule with feedback on how your Customers ultimately behave. For example, if a Customer is eventually flagged as fraudulent, you might want to see whether your Rules would have caught them in advance, or whether changes to your Rules might be more effective at identifying true-positive fraudulent Customers.

Our Feedback API lets you:

Assign feedback flags to your Customers to flag “bad users”
Place Customers into Test Sets to analyze performance of Rules and Workflows

Adding Feedback Labels to Customer records

Each Customer has a unique Customer ID. You can find the ID for any Customer by visiting the Customers Tab and clicking on their Customer Profile. You’ll find their Customer ID in the top left.

Using this customer ID, you can then make a PUT request to the Customer API to add labels. You can add them to one of three testsets by setting their testSetId to 1, 2, or 3

Ff this user has exhibited fraudulent behavior, you can also set their feedback to FLAGGED. See the example request in Postman below.

If you refresh your Customer Profile page you’ll now see the bottom of the Additional Data panel update to include the testsets and feedback

Backtesting Rules using Feedback and Test Sets

Once you have some customers with Feedback labels in a test set, you can run a test set through an updated version of a rule to see the impact on performance within the test set.

Open a workflow, and make a change to an IF / THEN rule. Click on the testing panel on the Right, and turn on the “Show Feedback” toggle. Choose a Rule Firing backtest, and then pick a Test Set from the drop down.

Using Feedback and Test Sets on the Analytics Tab

From the Analytics Tab, click Segmentation. At the top, you’ll see a few Filters for your analysis, including a Testsets selector. From here, you can pick an individual Test Set to to see the performance on the Graphs below

If you want to filter on the feedback you added in step 1, you’ll find it as a property under the base object is for the workflow you’re analyzing (see example below).