Hi,
I was trying to use the new Import CSV feature using our internal MINIO S3 Store. However, when I try to run the command, it appears that CRDB is looking for AWS host?
pq: could not find s3 bucket’s region: RequestError: send request failed
caused by: dial tcp: lookup s3.amazonaws.com on 169.40.0.1:53: no such host
Would it be possible for the import to respect the S3 protocol without actually looking for AWS hosts?
Best,
Hi @ramkumarkb. It does seem like that should be possible. I’ve created an
issue so we can prioritize this, feel free to chime in with any additional
details: https://github.com/cockroachdb/cockroach/issues/19832
We’ve just merged support for this. The next alpha release should have it. We’d be interested in any feedback after you try it. Thanks for the suggestion. https://github.com/cockroachdb/cockroach/pull/19860
@mjibson Thank you so much for the fix. As soon as the Alpha release is available I will test it with our internal S3 Minio Server. A large number of CSV files today are stored there and it would be a good test for this feature.
On a slightly related topic, is there a way to ignore the 1st record while doing the CSV imports. I have large CSV files and am finding myself to remove the 1st record of these CSV before I can import them to CRDB.
Thanks again!
No, currently there’s no way to ignore the first row.
@ramkumarkb, out of curiosity, do you want to skip the first row because
it’s a header row or for some other reason?
@dan Yes, most of the export scripts from databases like Oracle, export CSV with the column names as the first row. I could not figure out how to use the comment option (in the Import) to achieve this.
Yeah, totally reasonable. I filed an issue for an option to skip a header
row in IMPORT CSV https://github.com/cockroachdb/cockroach/issues/19967