[Python] Pandas df to Apache iceberg table

Discussão em 'Python' iniciado por Stack, Fevereiro 27, 2025.

  1. Stack

    Stack Membro Participativo

    I'm trying to insert into a Iceberg table, but I'm getting issues due to the data types mismatching. I've pasted a part of my code. The error message I'm getting is : "errorMessage": "Schema change detected: {'new_columns': {}, 'modified_columns': {'low': 'decimal(13,10)', 'close': 'decimal(13,10)', 'open': 'decimal(13,10).'.....

    import json
    import awswrangler as wr
    import pandas as pd
    from decimal import Decimal, getcontext

    getcontext().prec = 30
    df['open'] = df['open'].apply(lambda x: Decimal(x).quantize(Decimal('0.0000000000')))
    df['high'] = df['high'].apply(lambda x: Decimal(x).quantize(Decimal('0.0000000000')))
    df['low'] = df['low'].apply(lambda x: Decimal(x).quantize(Decimal('0.0000000000')))
    df['close'] = df['close'].apply(lambda x: Decimal(x).quantize(Decimal('0.0000000000')))
    df['adjustedclose'] = df['adjustedclose'].apply(lambda x: Decimal(x).quantize(Decimal('0.0000000000')))

    #print(df.columns)
    print(df.dtypes)

    #print(df.head(10))

    wr.athena.to_iceberg(
    df=df,
    database='market_test',
    table='price',
    table_location='s3://blah',
    temp_path='s3://blah-temp',
    data_source = 'AwsDataCatalog'
    )


    Error message

    {
    "errorMessage": "Schema change detected: {'new_columns': {}, 'modified_columns': {'open': 'decimal(13,10)', 'high': 'decimal(13,10)', 'adjustedclose': 'decimal(13,10)', 'close': 'decimal(13,10)', 'low': 'decimal(13,10)', 'volume': 'bigint'}, 'missing_columns': {}}",
    "errorType": "InvalidArgumentValue",
    "requestId": "[]",
    "stackTrace": [
    " File \"/var/task/lambda_function.py\", line 40, in lambda_handler\n wr.athena.to_iceberg(\n",
    " File \"/opt/python/awswrangler/_config.py\", line 715, in wrapper\n return function(**args)\n",
    " File \"/opt/python/awswrangler/_utils.py\", line 178, in inner\n return func(*args, **kwargs)\n",
    " File \"/opt/python/awswrangler/athena/_write_iceberg.py\", line 434, in to_iceberg\n raise exceptions.InvalidArgumentValue(f\"Schema change detected: {schema_differences}\")\n"
    ]
    }```

    Continue reading...

Compartilhe esta Página