Gleam - distributed MapReduce for Golang

"First, generics are needed. Of course, we can use reflection. But it is noticeably slower, to the point that I do not want to show the performance numbers. Second, dynamic remote code execution is also needed if we want to dynamically adjust the execution plan. We could pre-build all the execution DAGs first and choose one of them during runtime. But it is very limiting. As everyone else here, I enjoyed the beauty of Go. How to make it work for big data?"

- Chris Lu

That's the right question.

Chris resolved the performance issues of reflection and the lack of Generics using a scripting language named LuaJIT. Rather than building the entire directed acyclic graph (DAG) and then choosing one branch during runtime, the scripting nature of LuaJIT allows dynamic remote code execution, allowing us to dynamically adjust the execution plan during runtime.