Cartesian product in Scala

val s  = Seq("1","2","3","4")
val t  = Seq("a","b","c","d","e","f")
val u  = Seq("x","y")
val v  = Seq("m","l","n","o","p")

val sq = Seq(s,t,u,v)

sq.foldLeft(Seq(""))((b,a) => b.flatMap{i=>a.map{j=>i+j}})


res9: Seq[String] = List(1axm, 1axl, 1axn, 1axo, 1axp, 1aym, 1ayl, 1ayn, 1ayo, 1ayp, 1bxm, 1bxl, 1bxn, 1bxo, 1bxp, 1bym, 1byl, 1byn, 1byo, 1byp, 1cxm, 1cxl, 1cxn, 1cxo, 1cxp, 1cym, 1cyl, 1cyn, 1cyo, 1cyp, 1dxm, 1dxl, 1dxn, 1dxo, 1dxp, 1dym, 1dyl, 1dyn, 1dyo, 1dyp, 1exm, 1exl, 1exn, 1exo, 1exp, 1eym, 1eyl, 1eyn, 1eyo, 1eyp, 1fxm, 1fxl, 1fxn, 1fxo, 1fxp, 1fym, 1fyl, 1fyn, 1fyo, 1fyp, 2axm, 2axl, 2axn, 2axo, 2axp, 2aym, 2ayl, 2ayn, 2ayo, 2ayp, 2bxm, 2bxl, 2bxn, 2bxo, 2bxp, 2bym, 2byl, 2byn, 2byo, 2byp, 2cxm, 2cxl, 2cxn, 2cxo, 2cxp, 2cym, 2cyl, 2cyn, 2cyo, 2cyp, 2dxm, 2dxl, 2dxn, 2dxo, 2dxp, 2dym, 2dyl, 2dyn, 2dyo, 2dyp, 2exm, 2exl, 2exn, 2exo, 2exp, 2eym, 2eyl, 2eyn, 2eyo, 2eyp, 2fxm, 2fxl, 2fxn, 2fxo, 2fxp, 2fym, 2fyl, 2fyn, 2fyo, 2fyp, 3axm, 3axl, 3axn, 3axo, 3axp, 3aym, 3ayl, 3ayn, 3ayo...

Scala : Another example of collectFirst

Say, we want to find the population of the native country of the first European bird in a list of birds. This can be done by flatmapping over the birds, thus ignoring the non-European birds, and then getting the head of the resulting list. But then, we have to go over each bird, although we need to just get the first bird. If the list is sufficiently large, this is very inefficient.

The other approach is to use find to get the first European bird, and then simply get its native country's population. But here we have to get the native country of the bird twice. In this example, that is cheap, but if this information requires an expensive database retrieval, then we may not want to do it twice. Where collectFirst becomes very important is when this operation is very expensive - imagine doing a statistical calculation that provides the value for the match.

case class Bird(name: String)


object PlayCollect {

  val birds = Seq(Bird("jaybird"), Bird("owl"), Bird("sparrow"))

  val nativeCountries  = Map( (Bird("jaybird")->"Malaysia"), (Bird("sparrow")->"Ireland"), (Bird("owl")->"Chile") )
  val continents = Map( ("Malaysia" -> "Asia"), ("Ireland" -> "Europe"), ("Chile" -> "South America") )
  val population = Map( ("Malaysia"->10000000), ("Ireland" -> 12000000), "Chile"->45000000 )

  val popEuBird = new PartialFunction[Bird, Int] {
    def apply(b: Bird) = {
      continents(nativeCountries(b)) match {
        case "Europe" => population(nativeCountries(b))
      }
    }
    def isDefinedAt(b: Bird) = {
      continents(nativeCountries(b)) match {
        case "Europe" => true
        case _ => false
      }
    }
  }

  def populationOfNativeCountryOfFirstEuropeanBird: Option[Int] = {
    birds collectFirst popEuBird
  }

  def main (args: Array[String]) {
    populationOfNativeCountryOfFirstEuropeanBird.foreach{pop => println(pop)}
  }
}

Scala : collectFirst example

On occasion, you want to find the first occurrence of an item in a list, then transform it to another type. Using partial functions with collectFirst, we can accomplish this.

collectFirst will first call isDefinedAt to determine if apply should be called for the current item in the list. Thus it will skip over the non matching elements in the list without incurring a match exception.

trait Animal
case class Mammal(name: String) extends Animal
case class Bird(name: String) extends Animal

val animals = Seq(Mammal("elephant"), Mammal("tiger"), Bird("raven"), Mammal("monkey"), Bird("peacock"), Bird("sparrow"))


    val matchBird = new PartialFunction[Animal, Bird] {
      def apply(p: Animal) = {
        p match {
          case b@Bird(name) => b
        }
      }

      def isDefinedAt(p: Animal) = {
          p match {
            case Bird(name) => true
            case _ => false
          }
      }
    }


    animals collectFirst matchBird

    res11: Option[Bird] = Some(Bird(raven))

Titan : using native hadoop libraries on MacOSX

Once you have built the native hadoop libraries on your MacOSX, you need to add this bit of code to bin/gremlin.sh so that it can find them:

if [ -e "${HADOOP_PREFIX}/lib/native/libhadoop.dylib" ]; then
   LIB_PATH="${HADOOP_PREFIX}/lib/native"
   if [ -n "${LD_LIBRARY_PATH:-}" ]; then
       LD_LIBRARY_PATH="${LIB_PATH}:${LD_LIBRARY_PATH}"     # For Linux
   else
       LD_LIBRARY_PATH="${LIB_PATH}"
   fi

   if [ -n "${DYLD_LIBRARY_PATH:-}" ]; then
       DYLD_LIBRARY_PATH="${LIB_PATH}:${DYLD_LIBRARY_PATH}" # For Mac
   else
       DYLD_LIBRARY_PATH="${LIB_PATH}"
   fi
fi

The only oddity here is that the script uses "set -u" at the top, which makes bash complain if you use uninitialized variables. So you have to append ":-" to the variables that you are testing. You can see that in the lines that test LD_LIBRARY_PATH etc.