Cockroach framework?

hello - we seem to be having some trouble getting deepstream.io to interface with cockroachdb. this is a bit surprising, considering that postgres is working, so i have been told.

is there any other framework that might already be suited for cockroachdb?

i am thinking there might be something similar to RethinkDB and Horizon.io ?

Hi!
Thanks for touching base with this. However at this point we really need some more input from you to help us help you. There’s much context between the lines here which we find hard to grasp. So let’s be technical:

  1. what is the interface issue that you’re having? Is there an error message? Are there postgres features that we need to emulate to make the adapter work?

  2. what do you call a “framework”? We are already trying to increase CockroachDB compatibility with multiple ORM frameworks, so we could point you to that as “other frameworks already suited for CockroachDB”, but I reckon this is not what you have in mind. Tell us about what you need and we can tell you more about what we can provide.

  3. What specific parts of RethinkDB and Horizon.io does your application need? For example my personal understanding is that what Horizon.io and RethinkDB have in common is the ability to run (part of) your application’s code by the database server and define database triggers. That’s also (partly) possible with postgres, so I am guessing this might be what you’re relying on. If that’s the case, then the good news is that we have those things on our roadmap, the bad news is that it’s going to take a little while more. But meanwhile we can’t really answer you clearly because we don’t know yet what features from RethinkDB/Horizon/pg your deepstream adapter really needs.

i have hired a guy in Ukraine to write a “deepstream connector” but i cant get any support on how to use it, or even if it works.

i thought somebody who is very familiar with cockroachDB internals was already working with the deepstream people? maybe that work stopped.

the last reply i received from the deepstream people (Wolfram H) was this:

The ones that work best are the ones that provide support for binary json - searcheable JSON docs a la Postgres or MySQL or pub/sub as Postgres provides

i should think that cockroachDB and deepstream.io would be a natural fit.

regarding other frameworks, when i look at meteor(mongo) and horizon(rethinkdb) i was just wondering if there was another framework that was well-suited for cockroachDB, but the more i use deepstream the more i appreciate it.

So have you tried your newly developed deepstream connector yourself?

If so, what do you see?

If not, how do you even know whether it exists?

hi - sorry to sound like a total idiot, but i dont even know what a WORKING deepstream connector is supposed to look like ! so no, i have not tried it myself. however, i am trying to push back on the deepstream people to develop it and list it on their webpage as an example, along with mongo, redis, rethinkdb, etc.

here is how we know it exists - it was “announced” on the deepstream slack page by the author who i hired:

https://www.npmjs.com/package/deepstream.io-storage-cockroachdb

You’re not a “total idiot”! I was really honestly trying to get you to state the obvious - what is the problem at hand and what this adapter is doing.

Now to be fair we didn’t really know there was a public announcement, so you providing this link is an extremely useful step forward in the discussion. I’ll have a brief look at the code.

I had a brief look at the source code, which is to be found here: https://github.com/perimetral/deepstream.io-storage-cockroachdb

The code is mostly good to go (there’s a minor issue which I have filed as an issue in the repo) and I do not see anything that would cause issues during use.

So assuming this adapter code is fine, why did you start this thread with the statement “we seem to be having some trouble getting deepstream to interface with cockroachdb”?

So far I can see the adapter exists, there is documentation online about how to use it, and I do not see any obvious mistakes that would cause it to fail. What was the reason for you to think there is “trouble”?

i thought there was trouble because of a couple of the comments on the deepstream slack page, particularly from Wolfram:

but deepstream doesn’t use relational constructs. In relational databases it’s basically just independent tables with two columns each - one for key, one for value. The ones that work best are the ones that provide support for binary json - searchable JSON docs a la Postgres or MySQL or pub/sub as Postgres provides

there were also a couple of comments between “perimetral” (the connector author i hired) and the deepstream community that made me think there may be issues. somebody used the word “trillers” and i was not sure what that was - it might have even been a misspelling.

but either way, i would be VERY HAPPY to see a working example using cockroach and deepstream. and again, i have never seen a deepstream connector work with any database, let alone cockroach.

it would be great if somebody from cockroachdb would talk to the deepstream people again and see if this can be finished (with a working example)

Do you have a public link to this Slack discussion? Or a transcript? I’d like to study this to see the technical arguments that were used.

if you give me your login name i can cut/paste my conversation with Wolfram and give you a bit more information

That doesn’t work, I do not have an account on their slack channel. That’s why I was asking for a public link or a transcript. Also we’d need to see the specific discussion that you’re referring to, it would be hard to know what to search for.

Oh nevermind I found how to log in!

knz - what is your slack login name?

It’s knz, and I found the relevant discussions. So here’s the deal:

  • your Ukrainian dev is doing the right thing. He’s made and polished your adapter, and he’s pushing to get it integrated into the deepstream repo. The deepstream guys gave him some homework regarding testing automation. I don’t understand what the deepstream guys want but (apparently) it has nothing to do with database/adapter compatibility, just integration and testing.
  • Wolfram has just highlighted that the deepstream/cockroachdb is currently going to be limited because we don’t provide native search for JSON data. That will come (eventually, later) and in the mean time your adapter may be slower than strictly necessary.
  • Wolfram seems supportive of this work and has referred to your adapter as a thing that will be announced to users.
  • There is talk that deepstream project will also release their own postgres adapter sometime soon and they’re suggesting to have a look at this for adapting to CockroachDB. However I don’t think this is a good idea because if I read between the lines they will rely on triggers and user-defined functions/procedures, and we don’t plan to support that any time soon. So your initiative to fund a CockroachDB-specific adapter was a good one.

So again I don’t think there are “troubles” and you may be in a good spot right now! Perhaps with just a little more patience things will resolve on their own, or if you think it’s taking too long ask for the precise steps needed to get your adapter integrated in the deepstream project.

As to how to use this adapter and determining whether it’s good enough for your application, unfortunately that’s where we can’t help more at this time. You’d need someone (perhaps a 3rd party) to download it from the aforementioned link and try it out on their own to evaluate how good it is.

ok thank you. i should think that cockroachdb also would have a great interest in this.

once this is working, i keep imagining combining it with something like express or vue and having a replacement for MEAN but with a database that can handle referential integrity.

Absolutely! Thanks for your support.

1 Like

FYI - got it working ! now i need to figure out exactly just what its doing. :wink:

the deepstream record is indeed being saved into a cockroach table, so i will declare this one a victory.

Excellent, that was fast!

1 Like

“fast” - that is either cruel or sarcastic. :wink: (sorry i had to say it)

No, that was because we had just had a conversation yesterday about this not working, and suddenly it works! Right in time for the holidays :slight_smile:

1 Like