How can I open this colossal .csv file?
I have an enormous 32.8GB .csv file containing patient vital data. I need to open it and sift through the entries, but am unable to.
The file is too large to open in Excel or Access. I've tried using applications that split .csv files into smaller pieces, but I don't think I have access to a machine with enough memory to handle the operation. Is there any way to handle this ridiculously big file?
- 4 weeks agoFavorite Answer
For something that large, you'd probably want something that read line by line instead of trying to open it all at once. It'd be trivial to do in Linux, but I've never tried working with such a large dataset in Windows.
If you're using Windows 10, enabling the Windows Subsystem for Linux is relatively easy. If you know what structure it uses, it'd then be relatively easy to sort into smaller files. For example
egrep "^A" LargeCSVFle.csv >> A.csv
would copy only lines starting with the letter A. Presumably, this may still be several gigabytes in size. If you find it's too large to open with your typical tools after that, you could narrow it further with something like
egrep "^A[a-n]" A.csv >> AaAn.csv
Sorry I can't offer a more native Windows solution.
- amania_rLv 74 weeks ago
On a Unix/Linux system you can use the split command. That can work with any size input file.
- Richard JLv 74 weeks ago
You can transfer it to a flash drive that has enough space to accept it, then you will be able to open it
- DeMoNsLaYeR575Lv 74 weeks ago
export your data in a format that is
1. hippa compliant (if you are in the USA) (exporting in a non secured CSV is in violation of HIPPA)
2. in small enough chunks to be able to be opened in excel
- How do you think about the answers? You can sign in to vote the answer.
- MarvinatorLv 74 weeks ago
Instead of using Excel to open the file, have you tried IMPORTING the file to Excel?
- Anonymous4 weeks ago
Are you using the 32-bit or 64-bit versions of Excel? If 64-bit, you should be able to open it.