|
|
Hej,
Up to what sizes are broadcast sets a good idea?
I have large dataset (~5 GB) and I'm only interested in lines with a
certain ID that I have in a file. The file has ~10 k entries.
I could either Join the dataset with the IDList or I could broadcast the ID
list and do the filtering in a Mapper.
What would be the better solution given the data sizes described above?
Is there a good rule of thumb when to switch from one solution to the other?
cheers Martin
|
Locked
