EOFError: Ran out of input using huggingface lib for training - deep-learning

Question

I am trying to fine-tune the donut model using the naver-clova-ix/cord-v2 dataset, but I get "EOFError: Ran out of input" issue. I have a windows based server with 2gpus, and tried to make only one visible for training by CUDA_VISIBLE_DEVICES=0. The downloaded data and .json and .arrow files for train/test/validation are not empty. Any idea?
>>train.py --config ../config/train_cord.yaml
resume_from_checkpoint_path: None
result_path: ./result
pretrained_model_name_or_path: naver-clova-ix/donut-base
dataset_name_or_paths:
- naver-clova-ix/cord-v2
sort_json_key: False
train_batch_sizes:
- 8
val_batch_sizes:
- 1
input_size:
- 1280
- 960
max_length: 768
align_long_axis: False
num_nodes: 1
seed: 2022
lr: 3e-05
warmup_steps: 300
num_training_samples_per_epoch: 800
max_epochs: 30
max_steps: -1
num_workers: 8
val_check_interval: 1.0
check_val_every_n_epoch: 3
gradient_clip_val: 1.0
verbose: True
exp_name: train_cord
exp_version: 20221103_132724
Config is saved at result\train_cord\20221103_132724\config.yaml
D:\Tools\PVPythonVE\Donut\lib\site-packages\pytorch_lightning\utilities\seed.py:49: LightningDeprecationWarning: `pytorch_lightning.utilities.seed.seed_everything` has been deprecated in v1.8.0 and will be removed in v1.10.0. Please use `lightning_lite.utilities.seed.seed_everything` instead.
"`pytorch_lightning.utilities.seed.seed_everything` has been deprecated in v1.8.0 and will be"
Global seed set to 2022
D:\Tools\PVPythonVE\Donut\lib\site-packages\torch\functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\TensorShape.cpp:3191.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Some weights of DonutModel were not initialized from the model checkpoint at naver-clova-ix/donut-base and are newly initialized because the shapes did not match:
- encoder.model.layers.0.blocks.1.attn_mask: found shape torch.Size([3072, 100, 100]) in the checkpoint and torch.Size([768, 100, 100]) in the model instantiated
- encoder.model.layers.1.blocks.1.attn_mask: found shape torch.Size([768, 100, 100]) in the checkpoint and torch.Size([192, 100, 100]) in the model instantiated
- encoder.model.layers.2.blocks.1.attn_mask: found shape torch.Size([192, 100, 100]) in the checkpoint and torch.Size([48, 100, 100]) in the model instantiated
- encoder.model.layers.2.blocks.3.attn_mask: found shape torch.Size([192, 100, 100]) in the checkpoint and torch.Size([48, 100, 100]) in the model instantiated
- encoder.model.layers.2.blocks.5.attn_mask: found shape torch.Size([192, 100, 100]) in the checkpoint and torch.Size([48, 100, 100]) in the model instantiated
- encoder.model.layers.2.blocks.7.attn_mask: found shape torch.Size([192, 100, 100]) in the checkpoint and torch.Size([48, 100, 100]) in the model instantiated
- encoder.model.layers.2.blocks.9.attn_mask: found shape torch.Size([192, 100, 100]) in the checkpoint and torch.Size([48, 100, 100]) in the model instantiated
- encoder.model.layers.2.blocks.11.attn_mask: found shape torch.Size([192, 100, 100]) in the checkpoint and torch.Size([48, 100, 100]) in the model instantiated
- encoder.model.layers.2.blocks.13.attn_mask: found shape torch.Size([192, 100, 100]) in the checkpoint and torch.Size([48, 100, 100]) in the model instantiated
- encoder.model.layers.3.blocks.1.attn_mask: found shape torch.Size([48, 100, 100]) in the checkpoint and torch.Size([12, 100, 100]) in the model instantiated
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Using custom data configuration naver-clova-ix--cord-v2-6daad2d1fa36191a
Found cached dataset parquet (C:/Users/s.yousefi/.cache/huggingface/datasets/naver-clova-ix___parquet/naver-clova-ix--cord-v2-6daad2d1fa36191a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
Using custom data configuration naver-clova-ix--cord-v2-6daad2d1fa36191a
Found cached dataset parquet (C:/Users/s.yousefi/.cache/huggingface/datasets/naver-clova-ix___parquet/naver-clova-ix--cord-v2-6daad2d1fa36191a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
D:\Tools\PVPythonVE\Donut\lib\site-packages\pytorch_lightning\trainer\connectors\accelerator_connector.py:447: LightningDeprecationWarning: Setting `Trainer(gpus=2)` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=2)` instead.
f"Setting `Trainer(gpus={gpus!r})` is deprecated in v1.7 and will be removed"
Using 16bit native Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
[rank: 0] Global seed set to 2022
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2
[W C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [kubernetes.docker.internal]:59749 (system error: 10049 - The requested address is not valid in its context.).
[rank: 1] Global seed set to 2022
D:\Tools\PVPythonVE\Donut\lib\site-packages\torch\functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\TensorShape.cpp:3191.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Some weights of DonutModel were not initialized from the model checkpoint at naver-clova-ix/donut-base and are newly initialized because the shapes did not match:
- encoder.model.layers.0.blocks.1.attn_mask: found shape torch.Size([3072, 100, 100]) in the checkpoint and torch.Size([768, 100, 100]) in the model instantiated
- encoder.model.layers.1.blocks.1.attn_mask: found shape torch.Size([768, 100, 100]) in the checkpoint and torch.Size([192, 100, 100]) in the model instantiated
- encoder.model.layers.2.blocks.1.attn_mask: found shape torch.Size([192, 100, 100]) in the checkpoint and torch.Size([48, 100, 100]) in the model instantiated
- encoder.model.layers.2.blocks.3.attn_mask: found shape torch.Size([192, 100, 100]) in the checkpoint and torch.Size([48, 100, 100]) in the model instantiated
- encoder.model.layers.2.blocks.5.attn_mask: found shape torch.Size([192, 100, 100]) in the checkpoint and torch.Size([48, 100, 100]) in the model instantiated
- encoder.model.layers.2.blocks.7.attn_mask: found shape torch.Size([192, 100, 100]) in the checkpoint and torch.Size([48, 100, 100]) in the model instantiated
- encoder.model.layers.2.blocks.9.attn_mask: found shape torch.Size([192, 100, 100]) in the checkpoint and torch.Size([48, 100, 100]) in the model instantiated
- encoder.model.layers.2.blocks.11.attn_mask: found shape torch.Size([192, 100, 100]) in the checkpoint and torch.Size([48, 100, 100]) in the model instantiated
- encoder.model.layers.2.blocks.13.attn_mask: found shape torch.Size([192, 100, 100]) in the checkpoint and torch.Size([48, 100, 100]) in the model instantiated
- encoder.model.layers.3.blocks.1.attn_mask: found shape torch.Size([48, 100, 100]) in the checkpoint and torch.Size([12, 100, 100]) in the model instantiated
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Using custom data configuration naver-clova-ix--cord-v2-6daad2d1fa36191a
Found cached dataset parquet (C:/Users/s.yousefi/.cache/huggingface/datasets/naver-clova-ix___parquet/naver-clova-ix--cord-v2-6daad2d1fa36191a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
Using custom data configuration naver-clova-ix--cord-v2-6daad2d1fa36191a
Found cached dataset parquet (C:/Users/s.yousefi/.cache/huggingface/datasets/naver-clova-ix___parquet/naver-clova-ix--cord-v2-6daad2d1fa36191a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
[rank: 1] Global seed set to 2022
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2
[W C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [kubernetes.docker.internal]:59749 (system error: 10049 - The requested address is not valid in its context.).
[W C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [kubernetes.docker.internal]:59749 (system error: 10049 - The requested address is not valid in its context.).
[W C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [kubernetes.docker.internal]:59749 (system error: 10049 - The requested address is not valid in its context.).
----------------------------------------------------------------------------------------------------
distributed_backend=gloo
All distributed processes registered. Starting with 2 processes
----------------------------------------------------------------------------------------------------
D:\Tools\PVPythonVE\Donut\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py:606: UserWarning: Checkpoint directory E:\Sahar\ILR_sahar\donut\result\train_cord\20221103_132724 exists and is not empty.
rank_zero_warn(f"Checkpoint directory {dirpath} exists and is not empty.")
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]
| Name | Type | Params
-------------------------------------
0 | model | DonutModel | 201 M
-------------------------------------
201 M Trainable params
0 Non-trainable params
201 M Total params
402.248 Total estimated model params size (MB)
Epoch 0: 0%| | 0/50 [00:00<?, ?it/s] win32
win32
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "D:\Tools\PVPython\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "D:\Tools\PVPython\lib\multiprocessing\spawn.py", line 115, in _main
preparation_data = reduction.pickle.load(from_parent)
EOFError: Ran out of input
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "D:\Tools\PVPython\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "D:\Tools\PVPython\lib\multiprocessing\spawn.py", line 115, in _main
preparation_data = reduction.pickle.load(from_parent)
EOFError: Ran out of input