pyspark.sql.DataFrame.fillna¶

DataFrame.fillna(value: Union[LiteralType, Dict[str, LiteralType]], subset: Union[str, Tuple[str, …], List[str], None] = None) → DataFrame[source]¶

Replace null values, alias for na.fill(). DataFrame.fillna() and DataFrameNaFunctions.fill() are aliases of each other.

New in version 1.3.1.

Changed in version 3.4.0: Supports Spark Connect.

Parameters

valueint, float, string, bool or dict: Value to replace null values with. If the value is a dict, then subset is ignored and value must be a mapping from column name (string) to replacement value. The replacement value must be an int, float, boolean, or string.
subsetstr, tuple or list, optional: optional list of column names to consider. Columns specified in subset that do not have matching data types are ignored. For example, if value is a string, and subset contains a non-string column, then the non-string column is simply ignored.

Returns

DataFrame: DataFrame with replaced null values.

Examples

>>> df = spark.createDataFrame([
...     (10, 80.5, "Alice", None),
...     (5, None, "Bob", None),
...     (None, None, "Tom", None),
...     (None, None, None, True)],
...     schema=["age", "height", "name", "bool"])

Fill all null values with 50 for numeric columns.

>>> df.na.fill(50).show()
+---+------+-----+----+
|age|height| name|bool|
+---+------+-----+----+
| 10|  80.5|Alice|NULL|
|  5|  50.0|  Bob|NULL|
| 50|  50.0|  Tom|NULL|
| 50|  50.0| NULL|true|
+---+------+-----+----+

Fill all null values with False for boolean columns.

>>> df.na.fill(False).show()
+----+------+-----+-----+
| age|height| name| bool|
+----+------+-----+-----+
|  10|  80.5|Alice|false|
|   5|  NULL|  Bob|false|
|NULL|  NULL|  Tom|false|
|NULL|  NULL| NULL| true|
+----+------+-----+-----+

Fill all null values with to 50 and “unknown” for ‘age’ and ‘name’ column respectively.

>>> df.na.fill({'age': 50, 'name': 'unknown'}).show()
+---+------+-------+----+
|age|height|   name|bool|
+---+------+-------+----+
| 10|  80.5|  Alice|NULL|
|  5|  NULL|    Bob|NULL|
| 50|  NULL|    Tom|NULL|
| 50|  NULL|unknown|true|
+---+------+-------+----+

pyspark.sql.DataFrame.explain

pyspark.sql.DataFrame.filter