[Python] Pyspark: how to extract hour from timestamp

Discussão em 'Python' iniciado por Stack, Maio 30, 2025.

  1. Stack

    Stack Membro Participativo

    I have a table like the following

    df

    +------------------------------------+-----------------------+
    |identifier |timestamp |
    +------------------------------------+-----------------------+
    |86311425-0890-40a5-8950-54cbaaa60815|2020-03-18 14:41:55 UTC|
    |38e121a8-f21f-4d10-bb69-26eb045175b5|2020-03-13 15:19:21 UTC|
    |1a69c9b0-283b-4b6d-89ac-66f987280c66|2020-03-16 12:59:51 UTC|
    |c7b5c53f-bf40-498f-8302-4b3329322bc9|2020-03-18 22:05:06 UTC|
    |0d3d807b-9b3a-466e-907c-c22402240730|2020-03-17 18:40:03 UTC|
    +------------------------------------+-----------------------+

    tmp.printSchema()
    root
    |-- identifier: string (nullable = true)
    |-- timestamp: string (nullable = true)


    I would like to have a column that take only the day and the hours from the timestamp.

    I am trying the following:

    from pyspark.sql.functions import hour
    df = df.withColumn("hour", hour(col("timestamp")))


    but I get the following

    +--------------------+--------------------+----+
    | identifier| timestamp|hour|
    +--------------------+--------------------+----+
    |321869c3-71e5-41d...|2020-03-19 03:34:...|null|
    |226b8d50-2c6a-471...|2020-03-19 02:59:...|null|
    |47818b7c-34b5-43c...|2020-03-19 01:41:...|null|
    |f5ca5599-7252-49d...|2020-03-19 04:25:...|null|
    |add2ae24-aa7b-4d3...|2020-03-19 01:50:...|null|
    +--------------------+--------------------+----+


    while I would like to have

    +--------------------+--------------------+-------------------+
    | identifier| timestamp|hour |
    +--------------------+--------------------+-------------------+
    |321869c3-71e5-41d...|2020-03-19 03:00:...|2020-03-19 03:00:00|
    |226b8d50-2c6a-471...|2020-03-19 02:59:...|2020-03-19 02:00:00|
    |47818b7c-34b5-43c...|2020-03-19 01:41:...|2020-03-19 01:00:00|
    |f5ca5599-7252-49d...|2020-03-19 04:25:...|2020-03-19 04:00:00|
    |add2ae24-aa7b-4d3...|2020-03-19 01:50:...|2020-03-19 01:00:00|
    +--------------------+--------------------+-------------------+

    Continue reading...

Compartilhe esta Página