Opentaps Search Feature - Revision history

Sichen at 18:45, 1 March 2010

2010-03-01T18:45:57Z

Sichen: /* Architecture */

2009-12-02T01:14:24Z

‎Architecture

Sichen: /* Architecture */

2009-11-05T21:47:18Z

‎Architecture

Sichen: /* Architecture */

2009-07-17T22:55:47Z

‎Architecture

Sichen at 22:49, 17 July 2009

2009-07-17T22:49:07Z

Jwickers: /* Architecture */

2009-07-08T20:19:03Z

‎Architecture

Sichen at 22:08, 26 June 2009

2009-06-26T22:08:51Z

Sichen: Protected "Opentaps Search Feature": Sysop page [edit=sysop:move=sysop]

2009-06-26T00:53:45Z

Protected "Opentaps Search Feature": Sysop page [edit=sysop:move=sysop]

Sichen: New page: This is a conceptual document and the features described here are being implemented TOC == Architecture == The basic idea is to use [https://www.hibernate.org/410.html Hibernate S...

2009-06-26T00:53:45Z

New page: This is a conceptual document and the features described here are being implemented __TOC__ == Architecture == The basic idea is to use [https://www.hibernate.org/410.html Hibernate S...

New page

This is a conceptual document and the features described here are being implemented

__TOC__

== Architecture ==

The basic idea is to use [https://www.hibernate.org/410.html Hibernate Search] to index data in the opentaps database, and then have a search tool which allows you to find results based on keywords rather than explicit parameters like on a find form. This is however different than [http://www.google.com/enterprise/gsa/ Google Enterprise Search] which is designed to index documents rather than database table values. However, the overall architecture should support integration to external search interfaces such as [http://www.google.com/enterprise/gsa/ Google Enterprise Search] fairly easily.

Now these are just my notes

<pre>

Each domain has a search capability: DomainSearch: ie --> AccountsSearch.searchByKeyword(...) -> List<Account>
SalesOpportunity.searchByKeyword(...) -> List<SalesOpportunity>

so each application (CRM, purchasing, etc.) can have a SearchService
which you can do setKeyWord(...); maybe some other set parameters.
then search() and then getAccounts() getContacts() getSalesOpportunities()

And then purchasing just has a search which allows you to getSuppliers() for now

the links should be part of the view layer not of the search interface so you could have search be used by different view apps, or not at all--just be part of internal app

hibernate search should be set to configure entities and fields and their weights
so for example, if I want SalesOpportunity.opportunityName to weigh 10x as much as SalesOpportunity.description
similar to applications/product/config/prodsearch.properties

the other thing we need is AOP on the delegator to index the entities as they are stored by delegator and do it in 1 transaction prior to ofbiz committing so we don't open up more transactions

here's a good thought experiment:
what if we integrate in Google Search Appliance which indexes your documents
so...let's say it's part of CRM and we want to link to documents
so we drop a new DocumentDomainSearchInterface/Impl into opentaps CRM
I guess what we need is a method in CRMSearchInterface/Impl which returns a link and a description for each result in the List<Document>
and then implement Google Search Appliance as one kind of DocumentDomainSearchInterface
</pre>

← Older revision		Revision as of 18:45, 1 March 2010
Line 35:		Line 35:

	The results of the search should be passed to our GWT EntityListViews so that the webapps could display them.		The results of the search should be passed to our GWT EntityListViews so that the webapps could display them.
		+
		+	== Clearing Search Indices ==
		+
		+	The hibernate search index files are located in <tt>runtime/lucene/indexes</tt> (this is defined in <tt>org.opentaps.domain.container.HibernateCfgGeneratorContainer.java</tt>) and can be cleared by
		+
		+	$ ant refresh
		+
		+	After clearing them, you will need to run the <tt>opentaps.createHibernateSearchIndex</tt> service again to re-build them.

@@ Line 5: / Line 5: @@
 The basic idea is to use [https://www.hibernate.org/410.html Hibernate Search] to index data in the opentaps database, and then have a search tool which allows you to find results based on keywords rather than explicit parameters like on a find form.  The overall architecture should support integration to external search interfaces such as [http://www.google.com/enterprise/gsa/ Google Enterprise Search] fairly easily.
-Search is a form of data retrieval similar to SQL.   Therefore, following the [[Domain Driven Architecture][, search should be implemented in each domain, so that it could be where the domain data is located.   One option is simply to implement search as a special method of a domain's repository,  but because keyword-based search is so different than SQL  queries, we have decided to separate them into specialized search services. Thus, each domain will have its own search service, defined in an interface and implemented in the domain implementation.  For example, the Party domain will have search services for parties, accounts, contacts, leads, and suppliers. The Order domain will have search services for quotes, sales opportunities, sales orders, and purchase orders.
+Search is a form of data retrieval similar to SQL.    To make it easy for you to switch one search service for another, we have created a search domain based on the  [[Domain Driven Architecture]].   All basic search functionality is implemented in this domain, even though they may the searching other domains.  In the search domain you would therefore find services for parties, accounts, contacts, leads, and suppliers as well as for quotes, sales opportunities, sales orders, and purchase orders.
-Each application (CRM, purchasing, etc.) can then have its own search service which  searches through several domains used by the application.  For example <tt>CrmsfaSearchService</tt> will access the account, contact, lead, order, sales opportunity, etc. search services.  It has method such as getAccounts(), getContacts(), getSalesOpportunities(), getSalesOrders() to return lists of entity objects.  Purchasing initially will just have search of the suppliers and orders domain to getSuppliers() and getPurchaseOrders().
+Each application (CRM, purchasing, etc.) can then have its own search service which reuses best search components from the common search domain.  For example <tt>CrmsfaSearchService</tt> will access the account, contact, lead, order, sales opportunity, etc. search services.  It has method such as getAccounts(), getContacts(), getSalesOpportunities(), getSalesOrders() to return lists of entity objects.  Purchasing initially will just have search of the suppliers and orders domain to getSuppliers() and getPurchaseOrders().
 Implementing search this way makes it easier to replace one search technology with another. For example, one day we may have a document management domain in opentaps, and there could be a DocumentSearchServiceInterface in this domain.   This  interface might specify that for a set of keywords, it will return a List<Document> of Document  objects which might have the names, summaries, and links to the actual documents. In one implementation, we might use the Google Search Appliance  to implement the DocumentDomainSearchInterface, but in another instance, we could use a  different document indexing and search tool.   Applications such as CRM and purchasing could then call on the document domain to search for documents, while being separated from the actual implementation of the indexing and search.

@@ Line 3: / Line 3: @@
 == Architecture ==
-The basic idea is to use [https://www.hibernate.org/410.html Hibernate Search] to index data in the opentaps database, and then have a search tool which allows you to find results based on keywords rather than explicit parameters like on a find form.  This is different than [http://www.google.com/enterprise/gsa/ Google Enterprise Search] which is designed to index documents rather than database table values.  However, the overall architecture should support integration to external search interfaces such as [http://www.google.com/enterprise/gsa/ Google Enterprise Search] fairly easily.
+The basic idea is to use [https://www.hibernate.org/410.html Hibernate Search] to index data in the opentaps database, and then have a search tool which allows you to find results based on keywords rather than explicit parameters like on a find form.  The overall architecture should support integration to external search interfaces such as [http://www.google.com/enterprise/gsa/ Google Enterprise Search] fairly easily.
-Search is a form of data retrieval similar to SQL.   Therefore, following the [Domain Driven Architecture], search should be implemented in each domain, so that it could be where the domain data is located.   One option is simply to implement search as a special method of a domain's repository,  but because keyword-based search is so different than SQL  queries, we have decided to separate them into specialized search services. Thus, each domain will have its own search service, defined in an interface and implemented in the domain implementation.  For example, the Party domain will have search services for parties, accounts, contacts, leads, and suppliers. The Order domain will have search services for quotes, sales opportunities, sales orders, and purchase orders.
+Search is a form of data retrieval similar to SQL.   Therefore, following the [[Domain Driven Architecture][, search should be implemented in each domain, so that it could be where the domain data is located.   One option is simply to implement search as a special method of a domain's repository,  but because keyword-based search is so different than SQL  queries, we have decided to separate them into specialized search services. Thus, each domain will have its own search service, defined in an interface and implemented in the domain implementation.  For example, the Party domain will have search services for parties, accounts, contacts, leads, and suppliers. The Order domain will have search services for quotes, sales opportunities, sales orders, and purchase orders.
-Each application (CRM, purchasing, etc.) can then have its own search service which  searches through several domains used by the application.  For example <tt>CrmsfaSearchService</tt> will access the account, contact, lead, order,  miss sales opportunity, etc. search services.  It has method such as getAccounts(), getContacts(), and getSalesOpportunities() to return lists of entity objects.  Purchasing  initially will just have search of the suppliers domain to getSuppliers().
+Each application (CRM, purchasing, etc.) can then have its own search service which  searches through several domains used by the application.  For example <tt>CrmsfaSearchService</tt> will access the account, contact, lead, order, sales opportunity, etc. search services.  It has method such as getAccounts(), getContacts(), getSalesOpportunities(), getSalesOrders() to return lists of entity objects.  Purchasing initially will just have search of the suppliers and orders domain to getSuppliers() and getPurchaseOrders().
-Implementing search this way makes it easier to replace one search technology with another. For example, one day we may have a document management domain in opentaps, and there could be a DocumentSearchServiceInterface in this domain.   This  interface might specify that for a set of keywords, it will return a List<Document> of Document  objects which might have the names, summaries, and links to the actual documents. In one implementation, we might use the Google Search Appliance  to implement the DocumentDomainSearchInterface, but in another instance, we could use a  different document indexing and search tool.   Applications such as CRM and purchasing could then call on the document domain to search for documents, while being separated from the actual implementation of the indxing and search.
+Implementing search this way makes it easier to replace one search technology with another. For example, one day we may have a document management domain in opentaps, and there could be a DocumentSearchServiceInterface in this domain.   This  interface might specify that for a set of keywords, it will return a List<Document> of Document  objects which might have the names, summaries, and links to the actual documents. In one implementation, we might use the Google Search Appliance  to implement the DocumentDomainSearchInterface, but in another instance, we could use a  different document indexing and search tool.   Applications such as CRM and purchasing could then call on the document domain to search for documents, while being separated from the actual implementation of the indexing and search.
 == Configuration ==

@@ Line 7: / Line 7: @@
 Search is a form of data retrieval similar to SQL.   Therefore, following the [Domain Driven Architecture], search should be implemented in each domain, so that it could be where the domain data is located.   One option is simply to implement search as a special method of a domain's repository,  but because keyword-based search is so different than SQL  queries, we have decided to separate them into specialized search services. Thus, each domain will have its own search service, defined in an interface and implemented in the domain implementation.  For example, the Party domain will have search services for parties, accounts, contacts, leads, and suppliers. The Order domain will have search services for quotes, sales opportunities, sales orders, and purchase orders.
-Each application (CRM, purchasing, etc.) can then have its own search service which  searches through several domains used by the application.  For example ''CrmsfaSearchService'' will access the account, contact, lead, order,  miss sales opportunity, etc. search services.  It has method such as getAccounts(), getContacts(), and getSalesOpportunities() to return lists of entity objects.  Purchasing  initially will just have search of the suppliers domain to getSuppliers().
+Each application (CRM, purchasing, etc.) can then have its own search service which  searches through several domains used by the application.  For example <tt>CrmsfaSearchService</tt> will access the account, contact, lead, order,  miss sales opportunity, etc. search services.  It has method such as getAccounts(), getContacts(), and getSalesOpportunities() to return lists of entity objects.  Purchasing  initially will just have search of the suppliers domain to getSuppliers().
 Implementing search this way makes it easier to replace one search technology with another. For example, one day we may have a document management domain in opentaps, and there could be a DocumentSearchServiceInterface in this domain.   This  interface might specify that for a set of keywords, it will return a List<Document> of Document  objects which might have the names, summaries, and links to the actual documents. In one implementation, we might use the Google Search Appliance  to implement the DocumentDomainSearchInterface, but in another instance, we could use a  different document indexing and search tool.   Applications such as CRM and purchasing could then call on the document domain to search for documents, while being separated from the actual implementation of the indxing and search.

@@ Line 1: / Line 1: @@
 __TOC__
 == Architecture ==
-The basic idea is to use [https://www.hibernate.org/410.html Hibernate Search] to index data in the opentaps database, and then have a search tool which allows you to find results based on keywords rather than explicit parameters like on a find form.  This is however different than [http://www.google.com/enterprise/gsa/ Google Enterprise Search] which is designed to index documents rather than database table values.  However, the overall architecture should support integration to external search interfaces such as [http://www.google.com/enterprise/gsa/ Google Enterprise Search] fairly easily.
+The basic idea is to use [https://www.hibernate.org/410.html Hibernate Search] to index data in the opentaps database, and then have a search tool which allows you to find results based on keywords rather than explicit parameters like on a find form.  This is different than [http://www.google.com/enterprise/gsa/ Google Enterprise Search] which is designed to index documents rather than database table values.  However, the overall architecture should support integration to external search interfaces such as [http://www.google.com/enterprise/gsa/ Google Enterprise Search] fairly easily.
-Now these are just my notes.
+Implementing search this way makes it easier to replace one search technology with another. For example, one day we may have a document management domain in opentaps, and there could be a DocumentSearchServiceInterface in this domain.   This  interface might specify that for a set of keywords, it will return a List<Document> of Document  objects which might have the names, summaries, and links to the actual documents. In one implementation, we might use the Google Search Appliance  to implement the DocumentDomainSearchInterface, but in another instance, we could use a  different document indexing and search tool.   Applications such as CRM and purchasing could then call on the document domain to search for documents, while being separated from the actual implementation of the indxing and search.
-Each search service implements a ''SearchServiceInterface'' :
+== Configuration ==
-Each domain has a search capability, ''AccountsSearch'' -> List<Account>, ''SalesOpportunitySearch'' -> List<SalesOpportunity> ... which can have domain specific search options.
+The indexing of entities for search is configured by <tt>hot-deploy/opentaps-common/config/entitysearch.properties</tt>.   You can configure an entity to be indexed:
-And each application (CRM, purchasing, etc.) can have a Search Service which give an application specific interface to search any of the application related domains.
+And you can also configure the weights of the entity's fields:
-For example ''CrmSearchService'' provides getAccounts() getContacts() getSalesOpportunities() and could provide getParties() which would return any CRM related parties (Accounts + Contacts ...), and purchasing just has a search which allows you to getSuppliers() for now.
+Sometimes you'll have to index  entities such as SalesOpportunityRole or PartyRole,  so that the search services can use them to determine the type of party and whether it is an account, contact, or lead.
-The results of the search should be passed to our GWT EntityListViews so that the webapps could display them.
+Note that view entities can not be indexed, since they are not actually stored in the database.
-Hibernate search should be set to configure entities and fields and their weights
+For data stored using the ofbiz entity engine, opentaps uses [http://en.wikipedia.org/wiki/Aspect-oriented_programming  aspect oriented programming] (AOP) to intercept the ofbiz delegator's Entity-Event-Condition-Action (EECA) sequence.   Normally, after the delegator stores a value, it will evaluate EECA's attached to the entity  to see if any services should be run.   opentaps has added an aspect to the ofbiz delegator, so that it will also index the entity stored.
-The other thing we need is AOP on the delegator to index the entities as they are stored by delegator and do it in 1 transaction prior to ofbiz committing so we don't open up more transactions
+==  Displaying Search Results ==
-Here's a good thought experiment:
+The results of the search should be passed to our GWT EntityListViews so that the webapps could display them.

@@ Line 1: / Line 1: @@
-  This is a conceptual document and the features described here are being implemented
+This is a conceptual document and the features described here are being implemented
 __TOC__
@@ Line 7: / Line 7: @@
 The basic idea is to use [https://www.hibernate.org/410.html Hibernate Search] to index data in the opentaps database, and then have a search tool which allows you to find results based on keywords rather than explicit parameters like on a find form.  This is however different than [http://www.google.com/enterprise/gsa/ Google Enterprise Search] which is designed to index documents rather than database table values.  However, the overall architecture should support integration to external search interfaces such as [http://www.google.com/enterprise/gsa/ Google Enterprise Search] fairly easily.
-  Now these are just my notes
+Now these are just my notes.
-<pre>
+Each search service implements a ''SearchServiceInterface'' :
-Each domain has a search capability: DomainSearch: ie --> AccountsSearch.searchByKeyword(...) -> List<Account>
+Each domain has a search capability, ''AccountsSearch'' -> List<Account>, ''SalesOpportunitySearch'' -> List<SalesOpportunity> ... which can have domain specific search options.
-so each application (CRM, purchasing, etc.) can have a SearchService
+And each application (CRM, purchasing, etc.) can have a Search Service which give an application specific interface to search any of the application related domains.
-And then purchasing just has a search which allows you to getSuppliers() for now
+For example ''CrmSearchService'' provides getAccounts() getContacts() getSalesOpportunities() and could provide getParties() which would return any CRM related parties (Accounts + Contacts ...), and purchasing just has a search which allows you to getSuppliers() for now.
-the results of the search should be passed to our GWT EntityListViews so that the webapps could display them.
+The results of the search should be passed to our GWT EntityListViews so that the webapps could display them.
-hibernate search should be set to configure entities and fields and their weights
+Hibernate search should be set to configure entities and fields and their weights
 so for example, if I want SalesOpportunity.opportunityName to weigh 10x as much as SalesOpportunity.description
 similar to applications/product/config/prodsearch.properties
-the other thing we need is AOP on the delegator to index the entities as they are stored by delegator and do it in 1 transaction prior to ofbiz committing so we don't open up more transactions
+The other thing we need is AOP on the delegator to index the entities as they are stored by delegator and do it in 1 transaction prior to ofbiz committing so we don't open up more transactions
-here's a good thought experiment:
+Here's a good thought experiment:
 what if we integrate in Google Search Appliance which indexes your documents
 so...let's say it's part of CRM and we want to link to documents
@@ Line 34: / Line 36: @@
 I guess what we need is a method in CRMSearchInterface/Impl which returns a link and a description for each result in the List<Document>
 and then implement Google Search Appliance as one kind of DocumentDomainSearchInterface

← Older revision	Revision as of 00:53, 26 June 2009
(No difference)

Opentaps Search Feature - Revision history

Sichen at 18:45, 1 March 2010

Sichen: /* Architecture */

Sichen: /* Architecture */

Sichen: /* Architecture */

Sichen at 22:49, 17 July 2009

Jwickers: /* Architecture */

Sichen at 22:08, 26 June 2009

Sichen: Protected "Opentaps Search Feature": Sysop page [edit=sysop:move=sysop]

Sichen: New page: This is a conceptual document and the features described here are being implemented __TOC__ == Architecture == The basic idea is to use [https://www.hibernate.org/410.html Hibernate S...

Sichen: New page: This is a conceptual document and the features described here are being implemented TOC == Architecture == The basic idea is to use [https://www.hibernate.org/410.html Hibernate S...