向pyspark Dataframe添加新行
问题内容:
是一个非常新的pyspark,但对熊猫很熟悉。我有一个pyspark数据框
# instantiate Spark
spark = SparkSession.builder.getOrCreate()
# make some test data
columns = ['id', 'dogs', 'cats']
vals = [
(1, 2, 0),
(2, 0, 1)
]
# create DataFrame
df = spark.createDataFrame(vals, columns)
想添加新的行(4,5,7),以便输出:
df.show()
+---+----+----+
| id|dogs|cats|
+---+----+----+
| 1| 2| 0|
| 2| 0| 1|
| 4| 5| 7|
+---+----+----+
问题答案:
正如bluephantom已经说过的那样,工会是要走的路。我只是在回答您的问题,以举一个pyspark示例:
# if not already created automatically, instantiate Sparkcontext
spark = SparkSession.builder.getOrCreate()
columns = ['id', 'dogs', 'cats']
vals = [(1, 2, 0), (2, 0, 1)]
df = spark.createDataFrame(vals, columns)
newRow = spark.createDataFrame([(4,5,7)], columns)
appended = df.union(newRow)
appended.show()
也请查看databricks常见问题解答:https://kb.databricks.com/data/append-a-row-to-rdd-or-
dataframe.html