We have a hive warehouse, and wanted to use spark for various tasks (mainly classification). At times write the results back as a hive table. For example, we wrote the following python function to find the total sum of original_table column two, grouped by original_table column one. The function works, but we are worried that it is inefficient, particularly the maps to convert to key-value pairs, and dictionary versions. Functions combiner, mergeValue, mergeCombiner are defined elsewhere, but work fine. from pyspark import HiveContext rdd = HiveContext(sc).sql('from original_table select *') #convert to key-value pairs key_value_rdd = rdd.map(lambda x: (x[0], int(x[1]))) #create rdd where rows are (key, (sum, count) combined = key_value_rdd.combineByKey(combiner, mergeValue, mergeCombiner) # creates rdd with dictionary values in order to create schemardd dict_rdd = combined.map(lambda x: {'k1': x[0], 'v1': x[1][0], 'v2': x[1][1]}) # infer the schema schema_rdd = HiveContext(sc).inferSchema(dict_rdd) # save schema_rdd.saveAsTable('new_table_name') Are there more efficient ways of doing the same thing? Continue reading...
