Pages

Monday, 7 August 2017

Customizing “q” Parameter in SOLR

 
Hi All,
         Today we are going to see how we are going to customize the Search parameter in solr. Almost all of our Projects will be having the Requirement of Customizing this parameter. Please follow the Below Instructions so you can change it easily.

package com.mycommercesearch.solr;
              
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.handler.component.SearchHandler;
import org.apache.solr.request.SolrQueryRequest;
import org.apache.solr.response.SolrQueryResponse;

public class SearchEndecaHanlder extends SearchHandler {
      
       public static String SEARCH="search";
      
       public static String QUERY_PARAM="q";

       public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp)   throws Exception{
              String queryParam=req.getParams().get(SEARCH);
              SolrParams paramSolrParams = req.getParams();
              NamedList<Object> nmList = new NamedList<Object>();
              nmList=paramSolrParams.toNamedList();
              nmList.remove(SEARCH);
              nmList.add(QUERY_PARAM, queryParam);
              paramSolrParams=SolrParams.toSolrParams(nmList);
              req.setParams(paramSolrParams);
              super.handleRequestBody(req, rsp);
       }
}

In this Scenario I have used "search" instead of  "q" and it worked out. Create a Jar File from the above class and paste it in <SOLR_ISTALED_DIR>solr-6.6.0\server\solr-webapp\webapp\WEB-INF\lib\

Registering the Custom Handler in Solr

Navigate to Solrconfig.xml and add the below entry.

<requestHandler name="/mysearch" class="com.mycommercesearch.solr.SearchEndecaHanlder">
    <lst name="defaults">
      <str name="echoParams">explicit</str>
      <int name="rows">10</int>
      <!-- <str name="df">text</str> -->
    </lst>
  </requestHandler>

Once if you restart and access with the Below Url and get the Results .

http://localhost:8983/solr/refrence/mysearch?search=bus&wt=json&df=refrence


Happy Searching !!!!

Exception Handling in Endeca

 
Hi All,
  I am Going to Share Some Basics note about Exception Handling in Endeca . We will be having Many Scenarios that has to be handled, so that the Users will not lead to seeing the Error or the Exceptions. I am sharing my Experience it varies from the Implementation Perspective from developers to developers .

When we are implementing the Assembler Approach there are only two types of Exceptions will be land in to and we have to handle everything within it.

Exception
Description
AssemblerException
This Exception states that the exception occurred during the process or creating the Assembler Request. Eg ene Query,Host Exception
CartridgeHandlerException
This Exception Occurs while calling the single Cartridge handler, usually exceptions like NPE,Range Exceptions .

Well, we have Seen These Two Exceptions and next step is to how to bring these Exceptions in to our applications. When the endeca contentItems are rendered from the JSP then it is with the Developer, who renders it and no need to worry about this status. Where else we are going with the Rest or the JAX-RS Approach which requires these Scenarios to be handled before Sending the Status to the Service.

When you see the Architecture, there is no way we can handled the Exception Thrown. So what we have to do? Customize the Invoke Assembler or Assembler Tools only for the Exceptions? Not required.

So here is what you have to do, When the Assembler Exception Happens or the Cartridge Handler Exceptions Happens you are able to see the @error Param on the Content Item, when you go in to this param , you could able to see the type of exception. You can write a wrapper on top of the Calling type method for example in the Actor Chains, if it is going to be the Rest MVC architecture and Nucleus Resource Class in case of the JAX-RS Implementation, take the ContentItem that InvokeAssmbler Returns send it the Method and Parse it, weather it is network, Invalid key, NPE, Range, MDEX Down, Class Cast Exception and handle it appropriately based on the Type of Exception Status and Error Codes.. If the Content Item is null then blindly throw it as 500 Status .

I have handled in the above way Instead of throwing the Generic Exceptions, and it is working as Excepted. Hope you have understood the Easiest Way of handling the types of Exceptions in Endeca.

If you have any other Reference of handling it can Suggest Below.

Happy Learning !!!!!


Sunday, 6 August 2017

Tuning Weblogic Server - ATG Servers

 
Hi All,

     I Was Facing the Slowness in Starting my weblogic server, I cannot point out the Exact Issue then googled it found the below solutions .

  Adding -Djava.security.egd=file:///dev/urandom or -Djava.security.egd=file:/dev/./urandom to JAVA_OPTIONS has helped me to solve the issue.

 Hope your weblogic is Starting Fastly.
Happy Learning !!!!!

Clearing Cache In Linux

 
Hi All,
         Today, I was Facing error with Respect to the Memory , Though I have Enough Memory in my Machine, it's not Clearing out . 

 How I identified this 

 1) Executed the Command free -m 

 this will give the Free Memory Report in the Linux Box.

 2) I am Running only one Instance but taking lot of memory ,it  has not cleared the Cache Memory this can be cleared out by the Below Command .

  #!/bin/sh
sync; echo 3 > /proc/sys/vm/drop_caches

 Make Sure you are running as the Root Before Executing this.

 Happy Learning !!!!

Implementing Partial Indexing in Endeca

 
Hi all,
    After Spending more time with the Partial Indexing, I am Writing this posts, so it could be useful for the other people trying to implement the Partial Indexing for their Projects.

Usually all the Projects, would have Configured the Baseline Indexing in the Scheduled Manner, If We have got the Requirement of Indexing the Inventory From Endeca , Will think of Having the Partial Indexing Configured for the Scheduled Manner, the Constraint is that it cannot pick up the Changes, Since the Inventory Comes or updates as part of the Feed Process.

Before pitching in to the Discussion of that, let’s see what are all can be achieved with the Partial Indexing.

When the Deployment is made from BCC by Creating a new Proj will update/add the Properties , if the Properties Belong to the Same Repository.ie Adding the Promotion will not reflect after the Deployment and the  Partial Indexing , the Reason is that the Promotion belongs to the Different Repository not the Product Catalog.

So , How Does the Flow Goes or what is Behind the Scenes of the Partial Indexing ? .
Before Knowing the Flow, we all should aware of the Basics of Partial Indexing.


Basics of Partial Indexing

There is a Repository called the Incremental Item Queue that is Responsible for Triggering the Partial Indexing. We have a Property called the Monitored OutputConfigs that should have our Config paths eg: ProductCatalogOutputConfig , Article OutputConfig.

There Will be Generation Id Associated With the Each OutputConfig Paths. So During Deployment the Sku or the Product, or Article Will be Added to this Incremental Item Queue Repository along with the Generation Id and the Item Descriptor . During the Partial Indexing this table will be Flushed out by submitting those to the Indexing . If the Products are not loading as part of the Partial Indexing after knowing that Items are avalaible in Incremental Item Queue Repository, then the Problem with miss match of the GenerationId.

You are also should be aware if the Tables from the DB Where these Operations take place.

srch_update_queue: this Table Where will be having the
config_repo_id will be the Config id of the output config path .
item_desc_name will be the Item Descriptor name .
repository_id is the Id of the updated product / Sku .
generation is the value of the Current id .
change_type is the value of the Change Type .
please find the Below Example .

values ('14500010','product','product101','2279','0');

srch_config this table is the Place where will be getting the Config Repoid and Generation for the Particular Config Path.

So now you are aware of the Flow of Partial Indexing Flow. If your Requirement to Indexing comes as part of the Feed then you have to identify the Changed Items and Write explicitly in to the Incremental Item Queue and Scheduled the Indexing of the Partial then it will take.

If your Property Belongs to the Different ItemDescriptior or Repository , make sure that you are making the Linking to any of the property in the Product Catalog  after linking only it will be taken as part of the Indexing otherwise will not be taken.
Only those Properties that will not be added as part of the BCC Deployment , We have to Write Explicitly in to the Incremental Item Queue. Others not Required .

So Keep this Information or the Constraint in mind Before Start Implementing to your Application .


Happy Learning.!!!!! 

Monday, 31 July 2017

Avoiding PMD Checkstyles

 
Hi Users,
       Most of us Will Face the Burden of Fixing the PMD Checkstyles for assuring the Quality to Client With Respect to Code. This Can be a time Bound Activity and Requires some prior Planning to avoid it. I am Sharing my Views on it Which gave me the Consisting Improvement in Results .

1) Always Good Habit of Defining the Constants in the Seperate files rather than Hard Coding it.

2)Always follow the Way of Defining the Variable .

3)Remove the Unused Code or the Dead Code.

4)Don't make the Logic Complex by Writing a Complex Loops.

5)Don't Write Too Long Method/Classes, Break them then and there as part of the Code Commit.

6)For any Repository Calls and other Calls use the Configurable Value , and Avoid it defining in the handler Levels .

7)Make Sure you Write the Tools Class whenever it is required , instead of creating in the Same place.

8)Make Sure that your Code SatisFies You, before you commit.



Happy Learning !!!!!

Defining multiple entity In Solr

 
Most of us when implementing the Search for the Site , the data we are going to process is not from the Same Table and same fields , for information on how to Index the Data From Database can be seen in my previous blog here. This deals only with data from multiple datasources or the data from different tables here.

Navigate to db-data-config .xml and edit it. I am going to setup the Customer Data for Search here.

<?xml version="1.0" encoding="UTF-8" ?>
<dataConfig>
<dataSource name="ds1" type="JdbcDataSource" driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost:3306/classicmodels" user="root" password="root"/>
<dataSource name="ds2" type="JdbcDataSource" driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost:3306/customerdata" user="root" password="root"/>

Here is the place where, I can define the different datasources . Here I have configured two types of datasources, one is called ds1 and another called as ds2. You can have different Set like hsql and XML also defined for processing.

<document>
   <entity name="products" dataSource="ds1" pk="id" query="select * from products" deltaImportQuery="select * from products"
   deltaQuery="select * from products where last_modified > '${dataimporter.last_index_time}'">
     <field column="productCode" name="id"/>
     <field column="productName" name="name"/>
     <field column="productDescription" name="description"/>
                 <field column="productLine" name="category"/>
    </entity>    
    
In the products entity, unique key is the id and we can mention the data source also here.         

   <entity name="customers" dataSource="ds2"  pk="customerNumber" query="select * from customers" deltaImportQuery="select * from customers"
   deltaQuery="select * from customers where last_modified > '${dataimporter.last_index_time}'">
     <field column="customerNumber" name="id"/>
     <field column="customerName" name="customerName"/>
     <field column="contactLastName" name="contactLastName"/>
                 <field column="contactFirstName" name="contactFirstName"/>      
                 <field column="phone" name="phone"/>
                 <field column="addressLine1" name="addressLine1"/>
                 <field column="addressLine2" name="addressLine2"/>
                 <field column="city" name="city"/>
                 <field column="state" name="state"/>
                 <field column="postalCode" name="postalCode"/>
                 <field column="country" name="country"/>
     <field column="salesRepEmployeeNumber" name="salesRepEmployeeNumber"/>
                 <field column="creditLimit" name="creditLimit"/>
  </entity>
</document>
</dataConfig>


If you are introducing the new entity it is must to have field called the id , which is used for the uniqueness of the records.

Querying for products

http://localhost:8983/solr/refrence/select?q=harley&wt=json&df=refrence

Querying for customers

http://localhost:8983/solr/refrence/select?q=gift&wt=json&df=customer

df is the data fields that holds this indexed data. Refrence is the datafield for products and customer is the datafield for customers.


Errors:

2017-04-22 10:24:17.256 WARN  (Thread-14) [   x:refrence] o.a.s.h.d.SolrWriter Error creating document : SolrInputDocument(fields: [category=Ships, id=S72_3212, name=Pont Yacht, description=Measures 38 inches Long x 33 3/4 inches High. Includes a stand.
Many extras including rigging, long boats, pilot house, anchors, etc. Comes with 2 masts, all square-rigged, _version_=1565373662235721728])
org.apache.solr.common.SolrException: [doc=S72_3212] missing required field: city


When you face the error you have to remove the field required=”true ” or make it “false” in managed-schema.xml as like below
                <field name="city" type="string" indexed="true" stored="true" required="false" multiValued="false" />

<field name="city" type="string" indexed="true" stored="true" multiValued="false" />


If you face below error

Solr Error Document is missing mandatory uniqueKey field id


It means your document does not have the property id which is defined like below in  <uniqueKey>id</uniqueKey> in managed-schema.xml


Happy learning !!!!

Faceting in Solr

 
Faceting can be defined as grouping up the grouping up the fields of search results, so that user can narrow down their search. Solr comes with simple implementation of it.




Parameter
Example
Explanation
facet
If this is set to true then facets are enable for the current search.
facet.field
Facets will be returned for these fields defined.
facet.prefix
This will return only the fields matching the prefix in the facets.
facet.contains
This will return the facets containing the term matching
facet.sort
This will sort the results based on the field given.
facet.limit
This will limit the facets to be returned.
facet.offset
This will display the facets from the given offset
facet.mincount
This will return the facets , having only the matching count.
facet.missing
This will return the facets that is matching query but not the facet matching
facet.method
Algorithm to be used .
facet.range
This has to be returned for range faceting

Standard Query Parser Parameters

 
QParameter Whatever we ask solr to search, it has to be denoted with the q Parameter. Once the Solr sees this parameter it will return the search results matching this parameter.

Usgae:
http://localhost:8983/solr/refrence/select?q=harley&wt=json


Specifying Terms for the Standard Query Parser A query to the standard query parser is broken up into terms and operators.
There are two types of terms:
Solr search for the single terms and phrases(combination of the words)

Wildcard Searches

Usually             Wildcard searches, searches for all the possible results, same concept is applicable in the solr as well.We can define in both the way

1)For the particular word we can define in the following way.

Usage:
http://localhost:8983/solr/refrence/select?q=te?ting&wt=json

When we define in the above way, then the solr searches for the words like texting and testing as well.


2) For All the possiblites

Usage:
http://localhost:8983/solr/refrence/select?q=test*&wt=json

When we define like this , solr will search for all the possible outcomes of the given word, like testing, tested etc.

Fuzzy Searches

Solr has some unique features, by which we can achieve the other terms for the searched term.

Usage:
http://localhost:8983/solr/refrence/select?q=seat~&wt=json

When we define the term like above it searches for the words like beat,feat etc.We can also define the edit distance so based on it will search for the terms .

Usage:
http://localhost:8983/solr/refrence/select?q=seat~1&wt=json

It will search for only feat,beat but not foat

Proximity Searches

Solr will search only in the specified distance between this two words.

Usage:
http://localhost:8983/solr/refrence/select?q=%221972%20GTA%22~10&wt=json

This will return results so it identifies it.

Where else

http://localhost:8983/solr/refrence/select?q=%221972%20GTA%22~0&wt=json

This will not return the results , since there are 5 words distance between it.

Range Searches

This will return only of the specified ranges results.

Usage:
http://localhost:8983/solr/refrence/select?q=id:{S18_2625%20toS18_2626}&wt=json


Boosting a Term with ^

We can use ^ for boosting of the term, and making it more relevant.

Usage:
http://localhost:8983/solr/refrence/select?q=Chopper^1%20harley&wt=json

Boolean Opertors in solr

AND (&&)

NOT (!)

OR (||)

+

-




Happy Learning !!!!!

Common Query Parameters in Solr

 
Following tutorials deals with Solr's common query parameters, which are supported by the Search
RequestHandlers

After the Data Setup it’s our time to learn about the Querying part of Solr. Solr comes with simple parameters for the Querying.

Parameter
Url Parameter Example
Explanation
 start
http://localhost:8983/solr/refrence/browse?q=ford&wt=json&start=4

When the start parameter is defined, solr start displaying the results from this .The Default value is 0. Setting the start parameter to some other number, such as 4, causes Solr to skip over the preceding records and start at the document identified by the offset.
rows
http://localhost:8983/solr/refrence/browse?q=harley&wt=json&rows=1
this is the row parameter, specifies the number of products should be returned in the results set.

fq (Filter Query)
http://localhost:8983/solr/refrence/browse?q=harley&wt=json&rows=2&fq=Knucklehead

http://localhost:8983/solr/refrence/browse?q=harley&wt=json&rows=2&fq=Knucklehead+Drag

this parameter is used for the filtering results from the results already returned. We can use the multiple instances of the fq parameter. We can also concanete this type of query as well.

fl (Field List)
http://localhost:8983/solr/refrence/browse?q=harley&wt=json&rows=2&fq=Knucklehead+Drag&fl=name

this field list parameter is used for explicitly explaning the solr to search only these fields. It is always good practice to define this field when we have more fields in our indexing data.Only the mentioned fields will be returned.
 Debug
http://localhost:8983/solr/refrence/browse?q=harley&wt=json&rows=2&fq=Knucklehead+Drag&debug=true
the debug parameter is used to see the debug information about the query.
explainOther
http://localhost:8983/solr/refrence/browse?q=harley&wt=json&rows=2&fq=Drag&debugQuery=on&explainOther=id:S10_4698
This is used to compare the results with the id:S10_4698

this query not only returns the debug information and also used to compare with the given value.
Wt
http://localhost:8983/solr/refrence/browse?q=harley&wt=json
The wt parameter selects the Response Writer that Solr should use to format the query's response.
omitHeader
http://localhost:8983/solr/refrence/browse?q=harley&wt=json&omitHeader=true
This parameter may be set to either true or false.
If set to true, this parameter excludes the header from the returned results. The header contains information
about the request, such as the time it took to complete. The default value for this   parameter is false.


 logParamsList

http://localhost:8983/solr/refrence/browse?q=harley&wt=json&omitHeader=true&logParamsList=q,fq

And only the 'q' and 'fq' parameters will be logged.
By default, Solr logs all parameters of requests. From version 4.7, set this parameter to restrict which parameters of a request are logged. This may help control logging to only those parameters considered important to your
organization.

 echoParams
http://localhost:8983/solr/refrence/browse?q=harley&wt=json&echoParams=explicit

The echoParams parameter controls what information about request parameters is included in the response header.
 Sort
http://localhost:8983/solr/refrence/browse?q=harley&wt=json&sort=id%20desc

http://localhost:8983/solr/refrence/browse?q=harley&wt=json&sort=id%20asc
The sort parameter arranges search results in either ascending (asc) or descending (desc) order.

Reference Application in Solr

 
Most of us before start learning any new technology, will want to see how this works, and should be visualized before starting it, so that the understanding or the learning will be clear. That’s why almost all of the vendors provide the Reference application for their product. This Reference application will allow us to have the strong fundamentals and good understanding of the product.
When I started learning the solr, I searched for this refrence application at that time , I did not get to know about this application , As the Day Passes by I get to know this application , so I wanted to share with you guys as well, so that you can also use it for your learning .

How to start this reference application?

Navigate to : <SOLR-Installed Dir>/solr-6.4.2/bin

Execute the below :  solr start -e techproducts

Once if you execute the above following operations will be happen automatically, ie indexing

Starting up Solr on port 8983 using command:
C:\Users\syedghouse14\Downloads\solr-6.4.2\bin\solr.cmd start -p 8983 -s "C:\Users\syedghouse14\Downloads\solr-6.4.2\example\techproducts\solr"
Solr is already setup and running on port 8983 with status:
{
  "solr_home":"C:\\Users\\syedghouse14\\Downloads\\solr-6.4.2\\example\\techproducts\\solr",
  "version":"6.4.2 34a975ca3d4bd7fa121340e5bcbf165929e0542f - ishan - 2017-03-01 23:30:23",
  "startTime":"2017-04-02T06:23:19.196Z",
  "uptime":"0 days, 0 hours, 0 minutes, 38 seconds",
  "memory":"29.5 MB (%6) of 490.7 MB"}
If this is not the example node you are trying to start, please choose a different port.
WARNING: Core 'techproducts' already exists!
Checked core existence using Core API command:
http://localhost:8983/solr/admin/cores?action=STATUS&core=techproducts
Solr techproducts example launched successfully. Direct your Web browser to http://localhost:8983/solr to visit the Solr Admin UI

Once it is started then access this application through URL: 

http://localhost:8983/solr/techproducts/browse



This Reference application has the following features .

  • TypeAhead Sugestions .
  • Global Search
  • Facet Selection/Deselction
  • Instock Facet
  • Category Selection
  • Submit / Reset
  • Simple Search
  • Spatial Search
  • Group By


You can play with the above functionalities.  You can refer my blog for more Interesting topics on Solr .  

Happy Learning !!!!!

Starting Solr In Debug mode

 
Make sure you imported your project in eclipse, consisting the solr folder and java project. You can maintain the folder structure as per your project requirement.

Create a bat file consisting of the following Contents in to it. And name it as solr-start.bat

cd "<Solr-Installed-Dir>\solr-6.4.2\bin"
solr.cmd start -a "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=18983"

once if you execute the above bat file then the solr server is started in the debug mode .
Configure the Below Setting in the Eclipse

Host:localhost
Port: 18983

As like below.


Make sure that you have Included all the libraries as below.


Put debug points and execute some query through the browser then you will get the debug in your eclipse .




Happy Learning!!!!! 
Subscribe to Newsletter
© Copyright 2015 Search Endeca. Designed by Bloggertheme9 | Distributed By Gooyaabi Templates. Powered by Blogger.