flatMap issue

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

flatMap issue

Punit Naik
I have a test file which has a json per line. When I do a flatMap on it, it
automatically splits the whole json line on every character. Why does this
happen?

So if I do:


val data=env.readTextFile("file:///home/punit/vik-in")
val j=data.flatMap { x=>x }
j.first(1).print()

This prints "{"

--
Thank You

Regards

Punit Naik
Reply | Threaded
Open this post in threaded view
|

Re: flatMap issue

Fabian Hueske-2
readTextFile reads a file line-wise.

Is it possible, that your first line only contains "{"?

2016-04-28 8:06 GMT+02:00 Punit Naik <[hidden email]>:

> I have a test file which has a json per line. When I do a flatMap on it, it
> automatically splits the whole json line on every character. Why does this
> happen?
>
> So if I do:
>
>
> val data=env.readTextFile("file:///home/punit/vik-in")
> val j=data.flatMap { x=>x }
> j.first(1).print()
>
> This prints "{"
>
> --
> Thank You
>
> Regards
>
> Punit Naik
>
Reply | Threaded
Open this post in threaded view
|

Re: flatMap issue

Punit Naik
No Sir, its one json per line.

On Thu, Apr 28, 2016 at 3:19 PM, Fabian Hueske <[hidden email]> wrote:

> readTextFile reads a file line-wise.
>
> Is it possible, that your first line only contains "{"?
>
> 2016-04-28 8:06 GMT+02:00 Punit Naik <[hidden email]>:
>
> > I have a test file which has a json per line. When I do a flatMap on it,
> it
> > automatically splits the whole json line on every character. Why does
> this
> > happen?
> >
> > So if I do:
> >
> >
> > val data=env.readTextFile("file:///home/punit/vik-in")
> > val j=data.flatMap { x=>x }
> > j.first(1).print()
> >
> > This prints "{"
> >
> > --
> > Thank You
> >
> > Regards
> >
> > Punit Naik
> >
>



--
Thank You

Regards

Punit Naik
Reply | Threaded
Open this post in threaded view
|

Re: flatMap issue

stefanobaghino
The behavior you described actually makes sense: by passing the identity
function (x => x) to flatMap, you're basically just flattening your data
set, and since in Scala strings are also a collection of characters, you
are presented with a collection of characters.
If you just one to do something on a line at a time without flattening the
result you just have to use map (e.g. data.map("-> " + _) will print out
each line preceded by an "arrow").
I hope I've been helpful.

On Thu, Apr 28, 2016 at 12:33 PM, Punit Naik <[hidden email]> wrote:

> No Sir, its one json per line.
>
> On Thu, Apr 28, 2016 at 3:19 PM, Fabian Hueske <[hidden email]> wrote:
>
> > readTextFile reads a file line-wise.
> >
> > Is it possible, that your first line only contains "{"?
> >
> > 2016-04-28 8:06 GMT+02:00 Punit Naik <[hidden email]>:
> >
> > > I have a test file which has a json per line. When I do a flatMap on
> it,
> > it
> > > automatically splits the whole json line on every character. Why does
> > this
> > > happen?
> > >
> > > So if I do:
> > >
> > >
> > > val data=env.readTextFile("file:///home/punit/vik-in")
> > > val j=data.flatMap { x=>x }
> > > j.first(1).print()
> > >
> > > This prints "{"
> > >
> > > --
> > > Thank You
> > >
> > > Regards
> > >
> > > Punit Naik
> > >
> >
>
>
>
> --
> Thank You
>
> Regards
>
> Punit Naik
>



--
BR,
Stefano Baghino

Software Engineer @ Radicalbit
Reply | Threaded
Open this post in threaded view
|

Re: flatMap issue

Punit Naik
Yes you have. Thanks a lot Stefano Sir!

On Thu, Apr 28, 2016 at 6:40 PM, Stefano Baghino <
[hidden email]> wrote:

> The behavior you described actually makes sense: by passing the identity
> function (x => x) to flatMap, you're basically just flattening your data
> set, and since in Scala strings are also a collection of characters, you
> are presented with a collection of characters.
> If you just one to do something on a line at a time without flattening the
> result you just have to use map (e.g. data.map("-> " + _) will print out
> each line preceded by an "arrow").
> I hope I've been helpful.
>
> On Thu, Apr 28, 2016 at 12:33 PM, Punit Naik <[hidden email]>
> wrote:
>
> > No Sir, its one json per line.
> >
> > On Thu, Apr 28, 2016 at 3:19 PM, Fabian Hueske <[hidden email]>
> wrote:
> >
> > > readTextFile reads a file line-wise.
> > >
> > > Is it possible, that your first line only contains "{"?
> > >
> > > 2016-04-28 8:06 GMT+02:00 Punit Naik <[hidden email]>:
> > >
> > > > I have a test file which has a json per line. When I do a flatMap on
> > it,
> > > it
> > > > automatically splits the whole json line on every character. Why does
> > > this
> > > > happen?
> > > >
> > > > So if I do:
> > > >
> > > >
> > > > val data=env.readTextFile("file:///home/punit/vik-in")
> > > > val j=data.flatMap { x=>x }
> > > > j.first(1).print()
> > > >
> > > > This prints "{"
> > > >
> > > > --
> > > > Thank You
> > > >
> > > > Regards
> > > >
> > > > Punit Naik
> > > >
> > >
> >
> >
> >
> > --
> > Thank You
> >
> > Regards
> >
> > Punit Naik
> >
>
>
>
> --
> BR,
> Stefano Baghino
>
> Software Engineer @ Radicalbit
>



--
Thank You

Regards

Punit Naik