Import CSV does not work with MINIO S3 Store

Hi,

I was trying to use the new Import CSV feature using our internal MINIO S3 Store. However, when I try to run the command, it appears that CRDB is looking for AWS host?

pq: could not find s3 bucket’s region: RequestError: send request failed
caused by: dial tcp: lookup s3.amazonaws.com on 169.40.0.1:53: no such host

Would it be possible for the import to respect the S3 protocol without actually looking for AWS hosts?

Best,

Hi @ramkumarkb. It does seem like that should be possible. I’ve created an
issue so we can prioritize this, feel free to chime in with any additional
details: https://github.com/cockroachdb/cockroach/issues/19832

We’ve just merged support for this. The next alpha release should have it. We’d be interested in any feedback after you try it. Thanks for the suggestion. https://github.com/cockroachdb/cockroach/pull/19860

@mjibson Thank you so much for the fix. As soon as the Alpha release is available I will test it with our internal S3 Minio Server. A large number of CSV files today are stored there and it would be a good test for this feature.

On a slightly related topic, is there a way to ignore the 1st record while doing the CSV imports. I have large CSV files and am finding myself to remove the 1st record of these CSV before I can import them to CRDB.

Thanks again!

No, currently there’s no way to ignore the first row.

@ramkumarkb, out of curiosity, do you want to skip the first row because
it’s a header row or for some other reason?

@dan Yes, most of the export scripts from databases like Oracle, export CSV with the column names as the first row. I could not figure out how to use the comment option (in the Import) to achieve this.

Yeah, totally reasonable. I filed an issue for an option to skip a header
row in IMPORT CSV https://github.com/cockroachdb/cockroach/issues/19967