[Python] How do I read multiple files in S3 from EMR using PySpark wildcard?

Discussão em 'Python' iniciado por Stack, Abril 9, 2025.

  1. Stack

    Stack Membro Participativo

    It's in log.gz form, so I'm reading it in text, and there are sometimes mixed dates in the date folder because of the time difference by region.

    my log filename sample in S3

    in s3:mydata/input/2025/04/08

    mydata_2025040801.log.gz
    mydata_2025040802.log.gz
    mydata_2025040803.log.gz
    mydata_2025040804.log.gz
    mydata_2025040805.log.gz
    mydata_2025040901.log.gz


    it doesn't work.

    def generate_data_source_floor(month, day):
    # 문자열 포맷팅을 통해 S3 데이터 소스 경로를 생성합니다.
    return f"s3://mydata/input/2025/{month:02d}/{day:02d}/mydata_202504{day:02d}*"

    def generate_data_source_cell(month, day):
    next_day = day + 1
    # 문자열 포맷팅을 통해 S3 데이터 소스 경로를 생성합니다.
    return f"s3://mydata/input/2025/{month:02d}/{next_day:02d}/mydata_202504{day:02d}*"


    def get_log_by_ip(month, day, output_uri, target_c_ip):

    with SparkSession.builder.appName("Extract Vehicle Log").getOrCreate() as spark:

    logger.error("### Spark Start 3 ###")

    logger.info('### Spark start')

    log_data = spark.read.text(generate_data_source_floor(month, day), generate_data_source_cell(month, day))

    logger.info('### Spark 2')

    Continue reading...

Compartilhe esta Página