问题描述
我有一个大的 DataFrame(大约 4e+07 行).
I have a large DataFrame (circa 4e+07 rows).
求和时,我得到 2 个明显不同的结果,无论我是在在列选择之前还是之后进行求和.
此外,类型从 float32 更改为到 float64,即使总数都低于 2**31
When summing it, I get 2 significantly different results whether I do the sum before or after the column selection.
Also, the type changes from float32 to float64 even though totals are all below 2**31
df[[col1, col2, col3]].sum() Out[1]: col1 9.36e+07 col2 1.39e+09 col3 6.37e+08 dtype: float32 df.sum()[[col1, col2, col3]] Out[2]: col1 1.21e+08 col2 1.70e+09 col3 7.32e+08 dtype: float64
我显然遗漏了一些东西,有人遇到过同样的问题吗?
I am obviously missing something, has anybody had the same issue?
感谢您的帮助.
推荐答案
使用 np.float32 相对于 np.float64 可能会丢失精度
You can lose precision with np.float32 relative to np.float64
np.finfo(np.float32) finfo(resolution=1e-06, min=-3.4028235e+38, max=3.4028235e+38, dtype=float32)
和
np.finfo(np.float64) finfo(resolution=1e-15, min=-1.7976931348623157e+308, max=1.7976931348623157e+308, dtype=float64)
一个人为的例子
df = pd.DataFrame(dict( x=[-60499999.315, 60500002.685] * int(2e7), y=[-60499999.315, 60500002.685] * int(2e7), z=[-60499999.315, 60500002.685] * int(2e7), )).astype(dict(x=np.float64, y=np.float32, z=np.float32)) print(df.sum()[['y', 'z']], df[['y', 'z']].sum(), sep=' ') y 80000000.0 z 80000000.0 dtype: float64 y 67108864.0 z 67108864.0 dtype: float32