
I am trying to run some code, but getting error:
'DataFrame' object has no attribute '_get_object_id'
The code:
items = [(1,12),(1,float('Nan')),(1,14),(1,10),(2,22),(2,20),(2,float('Nan')),(3,300), (3,float('Nan'))] sc = spark.sparkContext rdd = sc.parallelize(items) df = rdd.toDF(["id", "col1"]) import pyspark.sql.functions as func means = df.groupby("id").agg(func.mean("col1")) # The error is thrown at this line df = df.withColumn("col1", func.when((df["col1"].isNull()), means.where(func.col("id")==df["id"])).otherwise(func.col("col1")))
12 Answers
You can't reference a second spark DataFrame inside a function, unless you're using a join. IIUC, you can do the following to achieve your desired result.
Suppose that means
is the following:
#means.show() #+---+---------+ #| id|avg(col1)| #+---+---------+ #| 1| 12.0| #| 3| 300.0| #| 2| 21.0| #+---+---------+
Join df
and means
on the id
column, then apply your when
condition
from pyspark.sql.functions import when df.join(means, on="id")\ .withColumn( "col1", when( (df["col1"].isNull()), means["avg(col1)"] ).otherwise(df["col1"]) )\ .select(*df.columns)\ .show() #+---+-----+ #| id| col1| #+---+-----+ #| 1| 12.0| #| 1| 12.0| #| 1| 14.0| #| 1| 10.0| #| 3|300.0| #| 3|300.0| #| 2| 21.0| #| 2| 22.0| #| 2| 20.0| #+---+-----+
But in this case, I'd actually recommend using a Window
with pyspark.sql.functions.mean
:
from pyspark.sql import Window from pyspark.sql.functions import col, mean df.withColumn( "col1", when( col("col1").isNull(), mean("col1").over(Window.partitionBy("id")) ).otherwise(col("col1")) ).show() #+---+-----+ #| id| col1| #+---+-----+ #| 1| 12.0| #| 1| 10.0| #| 1| 12.0| #| 1| 14.0| #| 3|300.0| #| 3|300.0| #| 2| 22.0| #| 2| 20.0| #| 2| 21.0| #+---+-----+
I think you are using Scala API, in which you use (). In PySpark, use [] instead.
0ncG1vNJzZmirpJawrLvVnqmfpJ%2Bse6S7zGiorp2jqbawutJobHBrZmiDcoSOqbCsqJGnuG6wwK2Yn6qRorJuu8GjnJysXZ2utHnNqGSarKSntqPB055koJ2kYryjtsScq2ahlA%3D%3D