Deepstream.io connector

hello - i have been using deepstream.io with various projects with much success

so, i am considering adding a deepstream-cockroachDB connector to their ORM collection. they have pretty good instructions on how to write a DS ORM.

does anybody have any opinions on doing this before i begin? after reading these excellent instructions on interfacing with node, i almost wonder how necessary it is,

note - i would not be doing the work myself but rather hiring somebody.

EDIT: am i looking at this thing entirely wrong?? i am getting the impression that whatever can talk to posgresql can talk to cockroachDB - true ?

Having not used Deepstream, after reading through their docs, I don’t think Cockroach is a strong fit for it, at least at this point. Their solution seems to work best with caches, messaging queues, and Nosql document stores. From their page on MongoDB integration:

Since deepstream is designed to structure data in JSON blobs, identified by a primary key, a NoSQL database like MongoDB is more suitable compared to relational databases like MySQL.

Cockroach also isn’t as focused on real-time event streams as many of the other databases Deepstream connects to, and the realtime aspect seems to be a large selling point of Deepstream. Because of this, I don’t think creating an integration for Cockroach would be as trivial as you think.

In regards to your question in the edit:

i am getting the impression that whatever can talk to posgresql can talk to cockroachDB - true ?

Sort of… Cockroach has elected to implement the Postgres wire protocol rather than making a new one, so most of the drivers and connectors that work with Postgres in the language of your choice should be able to connect to a Cockroach node and execute commands. However, Cockroach’s SQL dialect does not currently and I don’t believe plans to ever have parity with that of Postgres, so things which rely on certain SQL functions or features, such as ORMs, likely will not work with Cockroach as they do with Postgres, if they work at all.

Before you get started, we’re actually having a conversation with the guys at deepstream.io about integrating with CockroachDB this week. We may be getting this work done without you having to hire somebody to do the work.

As to the question of whatever can talk to postgresql being able to also talk to CockroachDB: we’re getting closer to that being true, but there are plenty of areas still where there are crucial differences. Many ORMs which talk to PostgreSQL use features which we don’t support. Some of the most commonly used ones we’re working to support, but there are others we will likely not include. So the answer is it depends on what the application is asking the database to do. If it’s basic SQL, no problem. Once you get to PostgreSQL extensions, it probably won’t work. And there’s a spectrum of stuff in between.

1 Like

ok thanks - i think the deepstream people are a little ticked off at me, since they quit answering me. :wink:

please keep me informed - i am eager to learn the status of this.

from what little i have seen of cockroach, it seems to me that combining it with deepstream will create an incredibly powerful tool, maybe even a new stack architecture?

and then combine it with express, and you have something so powerful that its hard to imagine: an SQL database with a MEAN like stack. WOW !

(is it just me, or does Meteor now seem totally obsolete?)

hello - may i ask how the chat with the deepstream people went?

status? we are almost done with a MariaDB connector as was wondering about a cockroachdb connector.

Apologies for the delay in responding, this fell off my radar. The Deepstream individual I chatted with was amenable to a CockroachDB backend, though there was no concrete plan to get started on the work. Is your message indicating that you’re almost finished with a MariaDB backend for Deepstream? I would imagine that would look similar to a CockroachDB backend and could possibly be used as the basis for one.

hi - actually i contracted with somebody to write the MariaDB connector. its now working as far as i can tell, but i really am completely unfamiliar with how deepstream connectors are supposed to work, since i have never seen how they are supposed to work.

the contractor gave me a simple example so i could see it working, sort of.

unfortunately the deepstream people are very busy with their hub project and currently are not showing much interest presently in supporting databases.

i am very tempted to re-hire the contractor to have a go at a cockroachdb connector.

UPDATE: i just contacted the MariaDB connector author to see if he is interested in doing one for cockroach.

i have a price quote for a deepstream.io connector - my concern is that since i have yet to see how a connector is supposed to work, i am afraid the same will happen with cockroachdb.

if i were to have the deepstream connector written, is there anybody out there in cockroach-land who could put an example together of how the deepstrean connector works and make it available?

the cockroachDB connector was mentioned on the deepstream slack discussion.

hi - any activity with deepstream? i keep dreaming about combining the awesome power of deepstream with cockroachDB, and maybe having something like the MEAN stack, except of course with a real database rather than this mongodb crap.

yasser at deepstream is OK with me hiring somebody to write a cockroachDB connector - but i would appreciate a little more input from cockroachDB before proceeding.

The only activity on our end was a brief glance at the API Deepstream requires for backends. Actually writing a connector still hasn’t percolated up on our priority list. I’m OK with you hiring someone to write a CockroachDB connector as well. I imagine a CockroachDB connector will look a lot like the MariaDB connector.

we have started writing a deepstream connector. i strongly believe that
combining cockroachdb and deepstream will result in an incredibly powerful
tool, possibly a new mini-framework.

Thank you,

Mark Edwards

reminder to peter - you approved this! [deepstream connector] – i am going to try to pursuade the deepstream people to test it and maybe provide an example. (https://www.npmjs.com/package/deepstream.io-storage-cockroachdb)

Congrats on getting it done, @edwardsmarkf! I’ll share with the team in case they’ve missed your forum update.

thank you Jessica - actually i paid a guy to do it for me. but i keep imagining combining cockroach with deepstream to come up with something similar to MEAN, but with a real database like cockroachdb, not these fake nonsense databases like what MEAN uses (can i get sued for saying that in a public forum??)

@edwardsmarkf I am so glad to hear that you got deepstream working for you. Upwards and onwards!

actually i should clarify: i hired a guy to write a
deepstream–>cockroachDB connector. there seems to be some trouble getting
it to work properly and i am not sure what the problem is.

ideally, i will end up with a very simple example of a
deepstream–>cockroachDB interface, but i suspect we are some ways away
from that.

maybe somebody from cockroachDB help us finish ?

Thank you,

Mark Edwards

Copied from 2017-03-01 post

From @edwardsmarkf
i continue to beat my head against the provervial wall in my ongoing efforts to interface cockroachdb to deepstream.io3.

although i was able to get the deepstream-cockroachdb connector working relatively easily, i found it far too limited in what it could do, so i opted to write my own interface attempting to use real sql calls rather than just using the JSON column.

as i beat my head against the wall, i keep wondering and hoping maybe somebody else has taken up the challenge (with better results thus far) in a cockroachdb framework?

since rethinkdb has horizon.io1, and mongo has both mean & meteor, maybe cockroachdb has a framework by now, or something similar?

oddly enough, i am stuck on one last problem, that being how to tell deepstream about a deleted row. when i delete an object row, there is no easy way to tell what happened on the server side, since all the jScript diff utilities dont really do diff the way unix does when you diff a pair of files (and easily displaying deleted rows). yes, this is not a cockroachdb issue, i am just venting…

cc @dianasaur323 for response.