How To

PySpark Word2Vec – Lessons Learned – Part 2

The next setting that I had to change was the spark.rpc.message.maxSize. This was changed when I got the following error. Serialized task XXX:XXX was XXX bytes, which exceeds max allowed: spark.rpc.message.maxSize (XXX bytes). Consider increasing spark.rpc.message.maxSize or using broadcast variables for large values. This error comes if there is large data that is being exchanged […]