[Python] What is the data_vars argument of xarray.open_mfdataset doing?

I have two datasets with identical dimension names and shapes and I am trying to use xarray.open_mfdataset() to merge them into one dataset before opening them. You can drop this code in your own IDE if you want to play with it:

import xarray as xr
import numpy as np

# Create dataset one and export it somewhere on your computer
state = ['California', 'Texas', 'New York']
time_mn = np.arange(1, 61)
time_yr = np.arange(1, 6)

ds1 = xr.Dataset(
data_vars={
'prcp_mn': (['state', 'time_mn'], np.random.rand(len(state), len(time_mn))),
'prcp_yr': (['state', 'time_yr'], np.random.rand(len(state), len(time_yr))),
'temp_mn': (['state', 'time_mn'], np.random.rand(len(state), len(time_mn))),
'temp_yr': (['state', 'time_yr'], np.random.rand(len(state), len(time_yr))),
},
coords={
'state': (['state'], state),
'time_mn': (['time_mn'], time_mn),
'time_yr': (['time_yr'], time_yr)
}
)
ds1.to_netcdf('/Users/foo/Downloads/somefolder/ds1.nc')

# Create second dataset
time_mn = np.arange(61, 121)
time_yr = np.arange(6, 11)
ds2 = xr.Dataset(
data_vars={
'prcp_mn': (['state', 'time_mn'], np.random.rand(len(state), len(time_mn))),
'prcp_yr': (['state', 'time_yr'], np.random.rand(len(state), len(time_yr))),
'temp_mn': (['state', 'time_mn'], np.random.rand(len(state), len(time_mn))),
'temp_yr': (['state', 'time_yr'], np.random.rand(len(state), len(time_yr))),
},
coords={
'state': (['state'], state),
'time_mn': (['time_mn'], time_mn),
'time_yr': (['time_yr'], time_yr)
}
)
ds2.to_netcdf('/Users/foo/Downloads/somefolder/ds2.nc')


Printing each dataset individually yields this:

ds1 = xr.open_dataset('ds1.nc')
print(ds1)

<xarray.Dataset> Size: 4kB
Dimensions: (state: 3, time_mn: 60, time_yr: 5)
Coordinates:
* state (state) <U10 120B 'California' 'Texas' 'New York'
* time_mn (time_mn) int64 480B 1 2 3 4 5 6 7 8 9 ... 53 54 55 56 57 58 59 60
* time_yr (time_yr) int64 40B 1 2 3 4 5
Data variables:
prcp_mn (state, time_mn) float64 1kB 0.5697 0.6228 0.2913 ... 0.8979 0.4984
prcp_yr (state, time_yr) float64 120B 0.8213 0.08581 ... 0.3644 0.02278
temp_mn (state, time_mn) float64 1kB 0.09875 0.7982 ... 0.9709 0.3221
temp_yr (state, time_yr) float64 120B 0.513 0.6703 ... 0.7104 0.2327


ds2 = xr.open_dataset('ds2.nc')
print(ds2)

<xarray.Dataset> Size: 4kB
Dimensions: (state: 3, time_mn: 60, time_yr: 5)
Coordinates:
* state (state) <U10 120B 'California' 'Texas' 'New York'
* time_mn (time_mn) int64 480B 61 62 63 64 65 66 ... 115 116 117 118 119 120
* time_yr (time_yr) int64 40B 6 7 8 9 10
Data variables:
prcp_mn (state, time_mn) float64 1kB 0.9125 0.6096 0.3521 ... 0.7349 0.4133
prcp_yr (state, time_yr) float64 120B 0.3651 0.5097 ... 0.2814 0.9877
temp_mn (state, time_mn) float64 1kB 0.7658 0.922 0.2113 ... 0.7582 0.4592
temp_yr (state, time_yr) float64 120B 0.2494 0.7272 ... 0.8682 0.08946


I thought what would happen when using open_mfdataset() to concat & merge the two datasets into one would be that the data variables and their associated dimensions would stay consistent since they are identically structured and named between the two datasets. This is what I expected:

Data variables:
prcp_mn (state, time_mn) ....
prcp_yr (state, time_yr) ....
temp_mn (state, time_mn) ....
temp_yr (state, time_yr) ....


Instead this is what I got:

Data variables:
prcp_mn (time_yr, state, time_mn) ....
prcp_yr (time_mn, state, time_yr) ....
temp_mn (time_yr, state, time_mn) ....
temp_yr (time_mn, state, time_yr) ....


After playing with the arguments of open_mfdataset(), I discovered that setting data_vars='minimal' or data_vars='different' both yield the result I was expecting:

ds = xr.open_mfdataset(paths=['ds1.nc', 'ds2.nc'], data_vars='minimal', decode_times=False)
print(ds)

<xarray.Dataset> Size: 7kB
Dimensions: (state: 3, time_mn: 120, time_yr: 10)
Coordinates:
* state (state) <U10 120B 'California' 'Texas' 'New York'
* time_mn (time_mn) int64 960B 1 2 3 4 5 6 7 ... 114 115 116 117 118 119 120
* time_yr (time_yr) int64 80B 1 2 3 4 5 6 7 8 9 10
Data variables:
prcp_mn (state, time_mn) float64 3kB dask.array<chunksize=(3, 60), meta=np.ndarray>
prcp_yr (state, time_yr) float64 240B dask.array<chunksize=(3, 5), meta=np.ndarray>
temp_mn (state, time_mn) float64 3kB dask.array<chunksize=(3, 60), meta=np.ndarray>
temp_yr (state, time_yr) float64 240B dask.array<chunksize=(3, 5), meta=np.ndarray>


If I supply the data_vars argument with a list of data variable names, it concatenates the "extra" dimension to just those specific data variables:

ds = xr.open_mfdataset(paths=['ds1.nc', 'ds2.nc'], data_vars=['prcp_mn', 'prcp_yr'], decode_times=False)
print(ds)

<xarray.Dataset> Size: 36kB
Dimensions: (state: 3, time_mn: 120, time_yr: 10)
Coordinates:
* state (state) <U10 120B 'California' 'Texas' 'New York'
* time_mn (time_mn) int64 960B 1 2 3 4 5 6 7 ... 114 115 116 117 118 119 120
* time_yr (time_yr) int64 80B 1 2 3 4 5 6 7 8 9 10
Data variables:
prcp_mn (time_yr, state, time_mn) float64 29kB dask.array<chunksize=(5, 3, 60), meta=np.ndarray>
prcp_yr (time_mn, state, time_yr) float64 29kB dask.array<chunksize=(60, 3, 5), meta=np.ndarray>
temp_mn (state, time_mn) float64 3kB dask.array<chunksize=(3, 60), meta=np.ndarray>
temp_yr (state, time_yr) float64 240B dask.array<chunksize=(3, 5), meta=np.ndarray>


My question is, what does the data_vars argument really mean, as I don't understand why the default value for the argument (all) doesn't yield what I was expecting. When I see the documentation refer to "data variables" to me that is talking about prcp_mn, prcp_yr, temp_mn, and temp_yr, not the dimensions of the data variables which really seems to be what is being concatenated when using the default data_vars='all' or specifying specific data variable names in the argument. Can someone give me some code examples that relate to what the values minimal and different in the data_vars argument are doing?

Continue reading...