Tuesday, 1 March 2016

PredictionIO Recommender Evaluation Tutorial

PredictionIO Recommender Evaluation Tutorial

 

Contents


This tutorial explains how to evaluate results generated by PredictionIO/template-scala-parallel-ecommercerecommendation. You can also see implementation details related to evaluations in PredictionIO framework.

Results


Results are Jaccard similarity and Precision@K values. If we repeat evaluation procedure we can see if our recommendation performance changes in time. Additionally evaluation loop helps by finding best algorithm parameters for our training data.


Resources

GitHub branch: https://github.com/goliasz/template-scala-parallel-ecommercerecommendation/tree/develpg
Tag: e0.7.0
PredictiomnIO official template page: http://templates.prediction.io/PredictionIO/template-scala-parallel-ecommercerecommendation

Step by Step Guide

I’m going to use Docker Image with PredictionIO to deploy recommendation engine and evaluate results. However there is no problem with testing evaluation algorithm in “standard” PredictionIO installation.

Let us run new docker container with preinstalled PredictionIO and Spark1.5.5. Before running container I’m going to create MyEngine folder which I will share with docker container as a volume. Recommendation engine will be deployed inside this folder and will be visible outside docker container. Even if I drop docker container my engine will stay untouched and I won’t loose my deployed PIO application.

I’m using Centos 6.7 box for my deployment. Let us create MyEngine folder inside my home directory.
mkdir MyEngine
Next let us run docker container with PredictionIO
docker run --hostname eval1 --name eval1 -it -v $HOME/MyEngine:/MyEngine goliasz/docker-predictionio /bin/bash
After running the docker container we are inside it with a bash root prompt.
root@eval1:/#
Now let us change folder to MyEngine. We are going to create PredictionIO engine inside.
root@eval1:/# cd MyEngine
root@eval1:/MyEngine#
Now let us create an recommendation engine. We have to pull right template from GitHub using “pio template get” command.
root@eval1:/MyEngine# pio template get goliasz/template-scala-parallel-ecommercerecommendation --version "e0.7.0" ecorec0
The command will create a new folder “ecorec0” which will contain recommendation engine implementation.

I’m giving following responses to the command. The responses are used to configure the engine. Scala sources inside will get package name according to my responses.
Please enter author's name: goliasz
Please enter the template's Scala package name (e.g. com.mycompany): com.kolibero
Please enter author's e-mail address: piotr.goliasz@kolibero.eu
Author's name:         goliasz
Author's e-mail:       piotr.goliasz@kolibero.eu
Author's organization: com.kolibero
Would you like to be informed about new bug fixes and security updates of this template? (Y/n) Y
Now we can start PredictionIO services.
root@eval1:/MyEngine# pio-start-all
After executing the command we can build the engine using “pio build --verbose”. First we have to change our current folder to new engine’s folder.
root@eval1:/MyEngine# cd ecorec0
Now we can build our engine.
root@eval1:/MyEngine/ecorec0# pio build --verbose
After the engine is built we have to register it in PIO framework.
root@eval1:/MyEngine/ecorec0# pio app new ecorec0
As a result we will get new application ID and Access Key. In my case application ID is 1. We will use Access Key to feed event store with test data.

Now we have to customize engine.json file with proper application name.
root@eval1:/MyEngine/ecorec0# vi engine.json
My engine.json after customization looks following. Changes are marked bold.
{
  "id": "default",
  "description": "Default settings",
  "engineFactory": "com.kolibero.ECommerceRecommendationEngine",
  "datasource": {
    "params" : {
      "appName": "ecorec0"
    }
  },
  "algorithms": [
    {
      "name": "ecomm",
      "params": {
        "appName": "ecorec0",
        "unseenOnly": false,
        "seenEvents": ["buy", "view"],
        "similarEvents": ["view"],
        "rank": 10,
        "numIterations" : 20,
        "lambda": 0.01,
        "seed": 3
      }
    }
  ]
}
At this point we need training data. Let us generate training data using python script delivered by template. We have to use Access Key to feed our event store with data.
root@eval1:/MyEngine/ecorec0# python data/import_eventserver.py --access_key VyhiNmp59j9qupci50M951IAqHsKVCvZXgMNhyn85crzbdaarYdz5OrnAY3JImxL
Note! Access key marked in bold is my value. You will have different value. You can check your application ID and access key using command “pio app list”.

At this point we have data in our event store and we are almost ready to start evaluation of the recommendation engine. The last this we need to do is a little customization of Evaluation.scala file and rebuilding the engine.

Let us open Evaluation.scala and change “INVALID_APP_NAME” into “ecorec0”. Changes have to be made at the end of the file.
root@eval1:/MyEngine/ecorec0# vi src/main/scala/Evaluation.scala
File before changes. Invalid strings marked in bold.
trait BaseEngineParamsList extends EngineParamsGenerator {
  protected val baseEP = EngineParams(
    dataSourceParams = DataSourceParams(
      appName = "INVALID_APP_NAME",
      evalParams = Some(DataSourceEvalParams(kFold = 2, queryNum = 5, buyTestScore = 10.0, viewTestScore = 1.0))))
}

object EngineParamsList extends BaseEngineParamsList {
  engineParamsList = for(
    rank <- Seq(10);
    numIterations <- Seq(20);
    lambda <- Seq(0.01))
    yield baseEP.copy(
      algorithmParamsList = Seq(
        ("ecomm", ECommAlgorithmParams("INVALID_APP_NAME", false, List("buy", "view"), List("view"), rank, numIterations, lambda, Option(3)))) )
}
File after changes. Changes in bold.
trait BaseEngineParamsList extends EngineParamsGenerator {
  protected val baseEP = EngineParams(
    dataSourceParams = DataSourceParams(
      appName = "ecorec0",
      evalParams = Some(DataSourceEvalParams(kFold = 2, queryNum = 5, buyTestScore = 10.0, viewTestScore = 1.0))))
}

object EngineParamsList extends BaseEngineParamsList {
  engineParamsList = for(
    rank <- Seq(10);
    numIterations <- Seq(20);
    lambda <- Seq(0.01))
    yield baseEP.copy(
      algorithmParamsList = Seq(
        ("ecomm", ECommAlgorithmParams("ecorec0", false, List("buy", "view"), List("view"), rank, numIterations, lambda, Option(3)))) )
}
Now let us rebuild the engine.
root@eval1:/MyEngine/ecorec0# pio build --verbose
At this point we can start evaluation code. Notice that I’m using “com.kolibero” package name in following command. If you have entered a different package name during engine registration then you have to use your own.   
root@eval1:/MyEngine/ecorec0# pio eval com.kolibero.RecommendationEvaluation com.kolibero.EngineParamsList
You should see a result similar to the following:
[INFO] [Jaccard] user: u2, jc: 0.09090909090909091, ars: 7, prs: 5
[INFO] [Jaccard] user: u2, jc: 0.3333333333333333, ars: 7, prs: 5
[INFO] [Jaccard] user: u4, jc: 0.09090909090909091, ars: 7, prs: 5
[INFO] [Jaccard] user: u4, jc: 0.3, ars: 8, prs: 5
[INFO] [Jaccard] user: u5, jc: 0.09090909090909091, ars: 7, prs: 5
[INFO] [Jaccard] user: u9, jc: 0.0, ars: 7, prs: 5
[INFO] [Jaccard] user: u10, jc: 0.08333333333333333, ars: 8, prs: 5
[INFO] [Jaccard] user: u3, jc: 0.0, ars: 7, prs: 5
[INFO] [Jaccard] user: u5, jc: 0.3, ars: 8, prs: 5
[INFO] [Jaccard] user: u9, jc: 0.2222222222222222, ars: 6, prs: 5
[INFO] [Jaccard] user: u7, jc: 0.1, ars: 6, prs: 5
[INFO] [Jaccard] user: u10, jc: 0.4, ars: 9, prs: 5
[INFO] [Jaccard] user: u1, jc: 0.0, ars: 7, prs: 5
[INFO] [Jaccard] user: u3, jc: 0.5, ars: 7, prs: 5
[INFO] [Jaccard] user: u6, jc: 0.0, ars: 7, prs: 5
[INFO] [Jaccard] user: u7, jc: 0.2, ars: 7, prs: 5
[INFO] [Jaccard] user: u8, jc: 0.08333333333333333, ars: 8, prs: 5
[INFO] [Jaccard] user: u1, jc: 0.2, ars: 7, prs: 5
[INFO] [Jaccard] user: u6, jc: 0.1, ars: 6, prs: 5
[INFO] [Jaccard] user: u8, jc: 0.3, ars: 8, prs: 5
[INFO] [MetricEvaluator] Iteration 0
[INFO] [MetricEvaluator] EngineParams: {"dataSourceParams":{"":{"appName":"ecorec0","evalParams":{"kFold":2,"queryNum":5,"buyTestScore":10.0,"viewTestScore":1.0}}},"preparatorParams":{"":{}},"algorithmParamsList":[{"ecomm":{"appName":"ecorec0","unseenOnly":false,"seenEvents":["buy","view"],"similarEvents":["view"],"rank":10,"numIterations":20,"lambda":0.01,"seed":3}}],"servingParams":{"":{}}}
[INFO] [MetricEvaluator] Result: MetricScores(0.16974747474747473,List(7.2, 0.22430555555555554))
[INFO] [CoreWorkflow$] Updating evaluation instance with result: MetricEvaluatorResult:
  # engine params evaluated: 1
Optimal Engine Params:
  {
  "dataSourceParams":{
    "":{
      "appName":"ecorec0",
      "evalParams":{
        "kFold":2,
        "queryNum":5,
        "buyTestScore":10.0,
        "viewTestScore":1.0
      }
    }
  },
  "preparatorParams":{
    "":{
    
    }
  },
  "algorithmParamsList":[{
    "ecomm":{
      "appName":"ecorec0",
      "unseenOnly":false,
      "seenEvents":["buy","view"],
      "similarEvents":["view"],
      "rank":10,
      "numIterations":20,
      "lambda":0.01,
      "seed":3
    }
  }],
  "servingParams":{
    "":{
    
    }
  }
}
Metrics:
  Jaccard (scoreThreshold=1.0): 0.16974747474747473
  PositiveCount (threshold=1.0): 7.2
  Precision@K (k=10, threshold=1.0): 0.22430555555555554
[INFO] [CoreWorkflow$] runEvaluation completed
At the end of the result you can see Jaccard metric together with positive count and Precision at K metric. These results will vary with your training data.

We have used for the evaluation RecommendationEvaluation and EngineParamsList. Both objects are declared inside Evaluation.scala file.

EngineParamsList

Declared in Evaluation.scala.

Contents:
trait BaseEngineParamsList extends EngineParamsGenerator {
  protected val baseEP = EngineParams(
    dataSourceParams = DataSourceParams(
      appName = "ecorec0",
      evalParams = Some(DataSourceEvalParams(kFold = 2, queryNum = 5, buyTestScore = 10.0, viewTestScore = 1.0))))
}

object EngineParamsList extends BaseEngineParamsList {
  engineParamsList = for(
    rank <- Seq(10);
    numIterations <- Seq(20);
    lambda <- Seq(0.01))
    yield baseEP.copy(
      algorithmParamsList = Seq(
        ("ecomm", ECommAlgorithmParams("ecorec0", false, List("buy", "view"), List("view"), rank, numIterations, lambda, Option(3)))) )
}
EngineParamList is inheriting on BaseEngineParamList. BaseEngineParamList is declaring basic parameters used by producing test data for evaluation. Test data are produced in readEval method declared in DataSource.scala class DataSource. Here we have following parameters:
  • kFold = 2,
  • queryNum = 5,
  • buyTestScore = 10.0,
  • viewTestScore = 1.0
kFold - training data taken from event strore will be folded and split into test and training data using modulo opertion with fold number and unique index number.
queryNum - evaluation queries will return 5 items
buyTestScore - score assigned to test data delivered to evaluation algorithm (buy events). See class DataSource.readEval.
viewTestScore - score assigned to test data delivered to evaluation algorithm (view events). See class DataSource.readEval.

EngineParamList declares evaluation loop. In this case w have a loop with only one engine params set. As we see in implementation of EngineParamList evaluation will be executed for following params set:
  • rank = 10
  • numIterations = 20
  • lambda = 0.01
  • unseenOnly = false
  • seenEvents = buy, view
  • similarEvents = view
  • seed = 3
We can implement a loop traversing various parameter sets by extending sequences in for loop declaration.
RecommendationEvaluation
Declared in Evaluation.scala.

object RecommendationEvaluation extends Evaluation {
  engineEvaluator = (
    ECommerceRecommendationEngine(),
    MetricEvaluator(
      metric = Jaccard(scoreThreshold = 1.0),
      otherMetrics = Seq(
        PositiveCount(scoreThreshold = 1.0),
        PrecisionAtK(k =10, scoreThreshold = 1.0)
      )
    )
  )
}

RecommendationEvaluation object is implementing what algorithms will be used for evaluation purpose. We can see here that main metric will use Jaccard class with parameter scoreThreshold = 1.0. Parameter with value 1.0 means that we will take into account all data delivered for testing purposes. We are assigning scores to test data inside BaseEngineParamsList in declaration of evalParams.  We have there:
evalParams = Some(DataSourceEvalParams(kFold = 2, queryNum = 5, buyTestScore = 10.0, viewTestScore = 1.0))))
This means that our data source of evaluation training data will produce data with kFold=2 (see DataSource.scala readEval method). We will use evaluation queries asking for 5 results. Buy event related item score will be 10.0. View event related item core will be 1.0.

We have secondary metric algorithms defined in RecommendationEvaluation. I have used PrecisionAtK and PositiveCount metric. Both metric also take parameters. See implementation of PrecisionAtK and PositiveCount. K cuts the number of predicted results and positiveCount is filtering events.

Evaluation Code

DataSource.scala

 

class DataSource, method readEval


Method preparing data training and test data for evaluation.

Training/test data preparation steps:
  1. Get all users - usersRDD variable. Data from $set events.
  2. Get all items - itemsRDD variable. Data from $set events.
  3. Get all events - eventsRDD variable. Data from “buy” and “view” events.
  4. Loop through 0 until kFold-1
    1. Split data into training and test events. Use modulo to split data.
    2. Prepare training view events - trainingViewEventsRDD variable
    3. Prepare training buy events - trainingBuyEventsRDD variable
    4. Prepare test view events - testViewEventsRDD variable
    5. Prepare test buy events - testBuyEventsRDD variable
    6. Create a list of test users. They will be used to ask queries by evaluation loop (testingUsers variable)
    7. Create test item scores consiting of user id, item id and score (testItemScores variable)
    8. Build resutl tuples containing training data, empty evaluation info, test queries+actual results list. See readEval declaration ( Seq[(TrainingData, EmptyEvaluationInfo, RDD[(Query, ActualResult)])] ).

Evaluation.scala

 

case class PrecisionAtK


Implements precisionAtK metric evaluating results of queries.

case class Jaccard


Implements Jaccard metric evaluating results of queries.
  def jaccardValue (A: Set[String], B: Set[String]) : Double = {
    return A.intersect(B).size.toDouble / A.union(B).size.toDouble
  }

case class PositiveCount


Implements PositiveCount metric evaluating results of queries.

object RecommendationEvaluation


Defines a list of evaluation metrics which will be used by evaluation.

trait BaseEngineParamsList


Defines a list of basic parameters used by preparation of test data for evaluation.

object EngineParamsList


Defines an evaluation loop. The loop creates engine parameter sets.
Posted by at
Labels: , , , ,