python - python ray-pyarrow.lib.ArrowInvalid：超出最大大小（2GB）(python ray - pyarrow.lib.ArrowInvalid: Maximum size exceeded (2GB))

Question

I am trying to load and process large files using ray.

(我正在尝试使用ray加载和处理大型文件。)

I am using ray for the purpose of multiprocessing the files and improving the speed of the solution.

(我将ray用于多文件处理和提高解决方案速度的目的。)

I keep running into this pyarrow error: pyarrow.lib.ArrowInvalid: Maximum size exceeded (2GB).

(我一直遇到这个pyarrow错误：pyarrow.lib.ArrowInvalid：超出最大大小（2GB）。)

It seems to have something to do with the plasma object store.

(似乎与等离子对象存储有关。)

I have tried to use huge_pages and mount it to the plasma store, increase the size of the ray object store on init.

(我试图使用huge_pages并将其安装到等离子存储中，以增加init上射线对象存储的大小。)

Any help would be great.

(任何帮助都会很棒。)

  ask by nicjaymo translate from so

Answer 1

等待大神答复