Datalog: does every value for a given entity attribute exactly match a given set?

General
1

Hi all. I asked this question on the Slack channel, but I thought this might be a better place for it.

As a setup, the model I’m working with has cards and persons. A person can be assigned to one or more cards, so I’ve got a cardinality-many attribute :card/assignees. The sample data I’m working with looks roughly like this (please excuse any syntax errors, this is more like pseudocode):

[{:db/id [:card/id "A"]   :card/assignees [[:person/id "a"]]}
 {:db/id [:card/id "B"]   :card/assignees [[:person/id "b"]]}
 {:db/id [:card/id "C"]   :card/assignees [[:person/id "c"]]}
 {:db/id [:card/id "AB"]  :card/assignees [[:person/id "a"] 
                                           [:person/id "b"]]}
 {:db/id [:card/id "ABC"] :card/assignees [[:person/id "a"] 
                                           [:person/id "b"] 
                                           [:person/id "c"]]}]

My requirement is that I have a filter for viewing cards by their assignees, so I’m getting a set of users from the front-end, and I need to return every card which is assigned to those users. This would be quite trivial if I needed to OR the users together, a la (filter-users #{"a" "b"}) => #{"A" "B" "AB" "ABC"}).

In my case, however, I need to AND the users together, so that (filter-users #{"a" "b"}) should return the cards that are assigned to both "a" and "b": the cards "AB" and "ABC".

I’ve been struggling to figure out how to do this in pure datalog. I did find a potential solution in this previous question, but involves calling out to the entity API to generate a set of values, and while I did manage to make the AND query work this way, it’s been difficult to compose this with other bits of the datalog I need for other requirements.

One of the fundamental problems I’m seeing is that if I pass in the target set into a query using collection binding, I’m only matching on a single member of the set at once, so any particular test to see whether a datom should be included in the result set has no access to the other members.

I have a feeling that there’s something fundamental I’m missing in my approach to the problem. Something about this stack overflow datalog answer seems like it gets close to the problem, but I’ve been unsuccessful translating it into its datomic equivalent.

1 Like
2

Can you give an example of “it’s been difficult to compose this with other bits of the datalog I need”? Note we also have clojure.core/set and all of datomic.api including d/q

3

Well, I suppose it’s more annoying than difficult. As one example, though, I’ve got another filter that should return true if the input set is a subset of all of the assignees plus a different card attribute, :card/author. So now in order to get the set-based stuff to work I need to extract the card author, get the set of assignees, (conj) the author into the assignees, and check whether the input set matches.

It’s not impossible, but doing all the set operations in datalog looks ugly to me, and I still have a feeling that this can be solved in pure datalog somehow without having to break out to the Clojure layer.

4

Thanks, okay I see and I agree with your summary. Here’s a relevant Slack chat from Rich (that does not answer your question): C-f “query style” http://www.dustingetz.com/:datomic-ion-launch-day-questions/

5
1 Like
6

This link appears broken. @dustingetz, do you happen to have an updated link?

7

Datomic Ion launch day questions answered by Rich

by Dustin Getz, http://www.hyperfiddle.net/

Ion functions vs Client API vs Entity API

dustingetz : @richhickey Does ion auto scaling apps essentially supersede the client model? Meaning there is no datomic client in play? Just a client-like local api

richhickey : @dustingetz right, client-free apps totally possible

dustingetz : Idiomatic, right?

richhickey : well, there’s zero reason for clients inside ions, but you may have legacy architecture that dictates use of clients. They’re not deprecated or anything
I think many apps can be written as ions
and of course you can mix and match

dustingetz : Are there technical reasons for no entity API or is it just historical at this point, and is the ion “local client api” equivalent in power to the entity api

richhickey : @dustingetz there are semantic differences with client (vs peer) that yield ‘no entity’. The ‘local’ API of ions matches the client API so people can do local REPL dev and testing (against cloud). So semantically, it’s compatible with ‘over wires’. That said, the perf of the same API in local mode within ions trounces clients. but keeping wire semantics means if you need to make an architectural shift that requires moving some parts to clients you’re not screwed
thus ‘client’ is the only API of cloud, ion or out

reduceclj : where does the performance gain comes from? eliminating the network calls?

richhickey : @reduceclj right, no wires, same process

Which query style is more idiomatic: Datalog functions or Ion functions?

dustingetz : Remote client requesting query for Ion cluster

1

[:find (pull ?f [:db/id *]) :where

2

[(partial contrib.datomic/datomic-entity-successors $) ?succ]

3

[(loom.alg-generic/bf-traverse ?succ :db/ident)]

Ion application local query

1

(->> (loom.alg-generic/bf-traverse

2

#(contrib.datomic/datomic-entity-successors $ %)

3

:db/ident)

4

(d/pull-many $ [:db/id *]))

if I am an Ion app, I can do both, is the former considered idiomatic? Because it is equivalent and works over wires

reduceclj : the former has the overhead of the network, which of the two you use depends on your needs/architecture

richhickey : @dustingetz the portability of the former seems like a flexibility win, but you may not care

dustingetz : @richhickey I like the former but that
thats a pretty wild “partial” in there
you would consider that idiomatic? Im ok with that if you are

richhickey : idioms take time to develop
these are some brand new power tools, would hate to pour concrete advice :slightly_smiling_face:

dustingetz : ok:)

Foreign datastore interop

val_waeselynck : To what extent is it reasonable to query other data stores (e.g a remote SQL or ElasticSearch server) from ions?

richhickey : @val_waeselynck you can do anything as long as you give the datomic cluster node’s role the necessary permissions

dustingetz : The ion will autoscale though and the foreign store will not
you’d need a queue

richhickey : your instances, your VPC

stuarthalloway : @dustingetz not everyone scales or needs to
but in any case the important point is, as @richhickey said, it is your instances, use them as you will

val_waeselynck : @richhickey thanks, it may be a good idea to make that obvious in the docs - it helps assess the power / limitations, especially since we’ve been accustomed to “be careful of the code Datomic runs for you” with tx fns

val_waeselynck : This is very exciting - it seems to give you the getting started experience of Firebase or Lambda with the scalability of advanced hand-rolled systems (edited)

richhickey : @val_waeselynck that still applies, you can hold up your txes

Do foreign datastore writes inside txfns slow down the transactor?

dustingetz : If I write to a durable kv store from a transactor fn and it takes 1ms to return, will this in practice slow down transactor throughput? Or are transactions processed in parallel up until the dynamo conditional put which i understand to be batched

richhickey : it’s going to be serial (edited)

dustingetz : Oh I see, because if there is a cas in the same tx, that needs a dbval

richhickey : because e.g. tx fns can query and expect to see prior txes
yes, stuff like that
but remember you will likely be initiating your txes from ions, so there’s another opportunity there for coordinated (if not transactional) work

dustingetz : I dont understand that, can you give me another hint

richhickey : put in elasticsearch, transact
means it might be in elasticsearch but tx fails, but not holding up txes

dustingetz : Oh ok, so if the other store supported two phase commit that might work too

richhickey : or whatever, cleanup on tx fail
you are likely retrying

dustingetz : thanks

reduceclj : why web ions and not just a ring app in a lambda function?

richhickey : the point is, you will be in your VPC, running in a role, able to do cool things
@reduceclj a) ease, b) not having to run in lambda execution container, c) running in db context vs making a client call, d) speed
the important value prop is API Gateway. Putting a lambda behind it is just one option (hint)

reduceclj : ok, pretty cool

Why is clojure.core/eval blacklisted in datalog evaluation?

dustingetz : @richhickey Since it is our instance, why is eval blacklisted in :where clause evaluation
Or is/could that be different in Ion (edited)

richhickey : there is an :accept list in ion, put stuff there to allow it

dustingetz : clojure.core/eval was excluded from cloud for security? security of your own jars or something

richhickey : nothing will run not on the list
i.e. nothing will be an entry point in tx/query/lambda
pants on by default

dustingetz : lol

richhickey : people inadvertently expose clients, query etc

dustingetz : Oh, ok makes sense

1 Like