[Python] pyspark dataframe schema, not able to set nullable false for csv files

I am trying to load csv file using pyspark. I am giving my own schema with columns nullable false, still when I print schema it shows them true. I checked the file data, there are no null entries for columns which are nullable false.

Code

from pyspark.sql.types import *

udemy_comments_file = '/Users/harbeerkadian/Documents/workspace/learn-spark/source_data/udemy/comments_spark.csv'
schema = StructType([StructField("id",StringType(),False),
StructField("course_id",StringType(),True),
StructField("rate",DoubleType(),True),
StructField("date",TimestampType(),True),
StructField("display_name",StringType(),True),
StructField("comment",StringType(),True),
StructField("new_id",StringType(),True)])
comments_df = spark.read.format('csv').option('header', 'true').schema(schema).load(udemy_comments_file)
comments_df.printSchema()
print("non null record count for id", comments_df.filter(comments_df.id.isNull()).count())


output

root
|-- id: string (nullable = true)
|-- course_id: string (nullable = true)
|-- rate: double (nullable = true)
|-- date: timestamp (nullable = true)
|-- display_name: string (nullable = true)
|-- comment: string (nullable = true)
|-- new_id: string (nullable = true)

non null record count for id 0


Ideally the id column nullable property should be false, as there are zero non null records.

Continue reading...