pyspark.sql.streaming.DataStreamWriter.options¶
-
DataStreamWriter.options(**options: OptionalPrimitiveType) → DataStreamWriter[source]¶ Adds output options for the underlying data source.
New in version 2.0.0.
Changed in version 3.5.0: Supports Spark Connect.
Notes
This API is evolving.
Examples
>>> df = spark.readStream.format("rate").load() >>> df.writeStream.option("x", 1) <...streaming.readwriter.DataStreamWriter object ...>
The example below specifies ‘numRows’ and ‘truncate’ options to Console source in order to print 3 rows for every batch without truncating the results.
>>> import time >>> q = spark.readStream.format( ... "rate").option("rowsPerSecond", 10).load().writeStream.format( ... "console").options(numRows=3, truncate=False).start() >>> time.sleep(3) >>> q.stop()