dedup

Algorithm

Find step

• indexName = {}
• Find list of unique name in entities
• for nameList in name
  • find entities with same name, n > 0
    • for entity in possible duplicates
      • compare authors
      • if mongeElkan > Threshold
        • indeName[name].push entities

Dedup

• for name in indexName
  • bestVersion = indexName[name][0]
  • bestVersion.identifier = bestIdentifier(map{$_.identifier})
  • bestVersion.authors = bestIdentifier(map{$_.identifier})
  • bestVersion.subjects = bestIdentifier(map{$_.identifier})
  • bestVersion.abstractText = bestIdentifier(map{$_.identifier})
  • bestVersion.blaspatialdingens = bestIdentifier(map{$_.identifier}) DB.replace …