I had a whole lot of large CSV files, totalling about 10.8 GB. All fields were delimited by speech marks (double-quotes). I wanted to load them into PostgreSQL using a COPY statement, but apparently, empty timestamp fields (looking like “”) weren’t of the right shape to be imported directly. I got an error message like this:

ERROR: invalid input syntax for type timestamp with time zone: ""
CONTEXT: COPY rawdata, line 218, column actual_off: ""

So I decided to get rid of all the speech marks. That’s OK, because there are no commas in any of the data … so I don’t know why the source would include them all anyway.

I started off with this:

sed 's/\"//g' data.csv > data-noquotes.csv

but it was terribly slow. After a bit of looking around, I got this:

cat data.csv | tr -d '"' > data-noquotes.csv

which was a whole lot faster.