IIS Index Server - Need help with Query using #filename and "OR"

Author

Need help with Query using #filename and "OR"

bstover@norcalmutual.com

2005-03-29, 8:48 pm

Indexing server running on Win2k3. All files are pdf -- almost 1
million in the archive. Adobe iFilter is installed and working. ASP
front end for searching. Code is below. I'm not much of an ASP
programmer, so any help someone could provide would be appreciated.

I can do free text queries for multiple filenames combined with free
text search that works fine -- like this:

<free text> AND (#filename=abc* OR #filename=def*)

However, when I JUST want to search for multiple filenames without the
free text portion of the query, the search pages hangs and eventually
times out. I've tried the following:

#filename=abc* OR #filename=def*
(#filename=abc* OR #filename=def*)
(#filename=abc*) OR (#filename=def*)

I know the file is indexed properly since I can find it like this:

#filename=<filename>

How do I construct this query??? Can someone help?

enumeration is set to TRUE. When I set to false, I get the message
that the query is too expensive.

Here is the main ASP search page:

****************************************
**************************
****************************************
**************************

<%@ Language=VBScript %>
<HTML>
<HEAD>
<TITLE>Official Documents Search - Start</TITLE>
<META name="VI60_DefaultClientScript" Content="VBScript">

<META NAME="GENERATOR" Content="Microsoft Visual Studio 6.0">
<SCRIPT ID=clientEventHandlersVBS LANGUAGE=vbscript>

</SCRIPT>
<meta http-equiv="Content-Type" content="text/html;
charset=iso-8859-1"></HEAD>
<body bgcolor="#ffffff" background="images2/HeaderBackground.gif"
vlink='#000080' leftmargin='0' topmargin='0' MARGINWIDTH="2"
MARGINHEIGHT="2" onLoad="java script:document.qform.polno.focus()">
<table width="200" border="0" align="left" cellpadding="0"
cellspacing="0">
<tr>
<td><img src="images2/Header_01.gif" width="72" height="178"></td>
<td><img src="images2/Header_02.gif" width="81" height="178"></td>
<td><img src="images2/Header_03.gif" width="194" height="178"></td>
<td><img src="images2/Header_04.gif" width="294" height="178"></td>
<td><img src="images2/Header_05.gif" width="130" height="178"></td>
</tr>
</table>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<p> </p>
<% Server.ScriptTimeout = 10 %>
<form name="qform" ACTION="x_FormString.asp" METHOD=get>
<table>
<tr>
<td rowspan="2"><img src="images2/option1.gif" width="150"
height="50"></td>
<td colspan="4" bgcolor="#99CCFF"><font size="2" face="Verdana,
Arial, Helvetica, sans-serif"><strong> Enter a Policy or Account
Number</strong></font></td>
<tr>
<td colspan="2"><font size="2" face="Verdana, Arial, Helvetica,
sans-serif">  </font><font size="2" face="Verdana, Arial,
Helvetica, sans-serif">
<input id=text1 name=polno >
</font></td>
<tr>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
</tr>
<tr>
<td> </td>
<td colspan="4" bgcolor="#99CCFF"><font size="2" face="Verdana,
Arial, Helvetica, sans-serif"><strong>Choose at least one of these
checkboxes to limit your query</strong></font>:</td>
</tr>
<tr>
<td width="150"> </td>
<td width="186"><font size="2" face="Verdana, Arial, Helvetica,
sans-serif">
<input type="checkbox" id=cbxDEC name=cbxDEC>
Dec Pages<br>
<input type="checkbox" id=cbxCOI name=cbxCOI>
COI  </font></td>
<td width="177"><font size="2" face="Verdana, Arial, Helvetica,
sans-serif">
<input type="checkbox" id=cbxNOC name=cbxNOC>
NOC<br>
<input type="checkbox" id=cbxINV name=cbxINV>
Invoice </font></td>
<td width="220"><font size="2" face="Verdana, Arial, Helvetica,
sans-serif">
<input type="checkbox" id=cbxMCL name=cbxMCL>
Material Change Letters<br>
<input type="checkbox" id=cbxBILL name=cbxBILL>
Billing Statement </font></td>
<td width="156"><font size="2" face="Verdana, Arial, Helvetica,
sans-serif">
<input type="checkbox" id=cbxEND name=cbxEND>
Endorsements <br>
</font></td>
</tr>
<tr>
<td> </td>
<td><font size="2" face="Verdana, Arial, Helvetica,
sans-serif"> </font></td>
<td>
<td></td>
<td>
</tr>
<tr>
<td> </td>
<td colspan="4" bgcolor="#99CCFF"><strong><font size="2"
face="Verdana, Arial, Helvetica, sans-serif">And/Or enter any other
text - i.e., Physician's Last Name, Invoice #, etc...
</font></strong></td>
</tr>
<tr>
<td> </td>
<td colspan="4"><font size="2" face="Verdana, Arial, Helvetica,
sans-serif">  </font><font size="2" face="Verdana, Arial,
Helvetica, sans-serif">
<input name=otherX id=text2 size="50" >
</font>
</td></td>
</tr>
<tr>
<td> </td>
<td colspan="4"> </td>
</tr>
<tr>
<td rowspan="2"><img src="images2/option2.gif" width="150"
height="50"></td>
<td colspan="4" bgcolor="#6699FF"><font size="2" face="Verdana,
Arial, Helvetica, sans-serif"><strong>Or perform a free text query -
This searches for any text on any page. </strong></font></td>
</tr>
<tr>
<td colspan="4"><font size="2" face="Verdana, Arial, Helvetica,
sans-serif">  </font> <font size="2" face="Verdana, Arial,
Helvetica, sans-serif">
<input name=FreeText id=freeText size="50" >
</font> </td>
<tr>
<td> </td>
<td colspan="4"><div align="center"><font size="2" face="Verdana,
Arial, Helvetica, sans-serif">
<INPUT id=submit1 type=submit value=Search name=submit1>
</font></div></td>
</table>
<font size="2" face="Verdana, Arial, Helvetica, sans-serif">

</font><br>
<br><br><br>
<INPUT id=text3 type="hidden" name=qstring style="WIDTH: 500px; HEIGHT:
22px" size=47>
<INPUT type ="hidden" id=text3 name=xxx value="<%=SearchString%>"
style="WIDTH: 367px; HEIGHT: 22px" size=47>
<input name="Advanced2" type=hidden checked>
</form>
</body>
</HTML>
****************************************
***************************
****************************************
***************************

Here is the parameter file

<%@LANGUAGE="VBSCRIPT" CODEPAGE="1252"%>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 3.0//EN" "html.dtd">
<HTML>
<HEAD>
<%
' ********** INSTRUCTIONS FOR QUICK CUSTOMIZATION **********
'
' This form is set up for easy customization. It allows you to modify
the
' page logo, the page background, the page title and simple query
' parameters by modifying a few files and form variables. The
procedures
' to do this are explained below.
'
'
' *** Modifying the Form Logo:

' The logo for the form is named is2logo.gif. To change the page logo,
simply
' name your logo is2logo.gif and place in the same directory as this
form. If
' your logo is not a GIF file, or you don't want to copy it, change the
following
' line so that the logo variable contains the URL to your logo.

FormLogo = "is2logo.gif"

'
' *** Modifying the Form's background pattern.

' You can use either a background pattern or a background color for
your
' form. If you want to use a background pattern, store the file with
the name
' is2bkgnd.gif in the same directory as this file and remove the remark
character
' the single quote character) from the line below. Then put the remark
character on
' the second line below.
'
' If you want to use a different background color than white, simply
edit the
' bgcolor line below, replacing white with your color choice.

' FormBG = "background = " & chr(34) & "is2bkgnd.gif" & chr(34)
FormBG = "bgcolor = " & chr(34) & "#FFFFFF" & chr(34)

' *** Modifying the Form's Title Text.

' The Form's title text is set on the following line.

%>
<TITLE>Official Documents Search - Results</TITLE>
<%
'
' *** Modifying the Form's Search Scope.
'
' The form will search from the root of your web server's namespace and
below
' (deep from "/" ). To search a subset of your server, for example,
maybe just
' a PressReleases directory, modify the scope variable below to list
the
' virtual path to search. The search will start at the directory you
specify
' and include all sub-directories.

FormScope = "/"

'
' *** Modifying the Number of Returned Query Results.
'
' You can set the number of query results returned on a single page
' using the variable below.

PageSize = 10
'
' You can set the total number of query results returned using the
' variable below.

MaxResults = 5000

'
' *** Setting the Locale.
'
' The following line sets the locale used for queries. In most cases,
this
' should match the locale of the server. You can set the locale below.

SiteLocale = "EN-US"

'
' *** Enable saving active searches in session variables
'
' The following line enables using session state to save multi-page
queries
' and avoid re-executing them when the user presses the Next Page
button.
' This should be set FALSE if sessions are disabled using
AspAllowSessionState.

UseSessions = FALSE
UseSessions = TRUE
Server.ScriptTimeout = 10

' ********** END QUICK CUSTOMIZATION SECTIONS ***********
%>

<LINK REL=STYLESHEET HREF="is2style.css" TYPE="text/css">

<META NAME="DESCRIPTION" CONTENT="Sample ASP query form for
Indexing Service">
<META NAME="AUTHOR" CONTENT="Indexing Service Team">
<META NAME="KEYWORDS" CONTENT="query, content, hit">
<META NAME="SUBJECT" CONTENT="sample form">
<META NAME="MS.CATEGORY" CONTENT="Internet">
<META NAME="MS.LOCALE" CONTENT="EN-US">
<META HTTP-EQUIV="Content-Type" CONTENT="text/html;
charset=iso-8859-1">
<%

xxx = ltrim(Request.QueryString("xxx"))
if xxx = "" or left(xxx,1) = " " THEN
Response.Redirect ("gen_error.asp")
End if
term = Request.QueryString("SearchString")
'esponse.Write session("xxx")

' Set Initial Conditions
NewQuery = FALSE
UseSavedQuery = FALSE
SearchString = ""

QueryForm = Request.ServerVariables("PATH_INFO")

' Did the user press a SUBMIT button to execute the form? If so get the
form variables.
if Request.ServerVariables("REQUEST_METHOD") = "POST" then
SearchString = Request.Form("SearchString")
SearchString = xxx

Advanced = Request.Form("Advanced")
' NOTE: this will be true only if the button is actually
pushed.
if Request.Form("Action") = "Go" then
NewQuery = TRUE
RankBase=1000
NextPageNumber = -1
end if
end if
if Request.ServerVariables("REQUEST_METHOD") = "GET" then
SearchString = Request.QueryString("qu")
Advanced = Request.QueryString("Advanced")
FormScope = Request.QueryString("sc")
RankBase = Request.QueryString("RankBase")
if Request.QueryString("pg") <> "" then
NextPageNumber = Request.QueryString("pg")
UseSavedQuery = UseSessions
NewQuery = not UseSessions
else
NewQuery = SearchString <> ""
NextPageNumber = -1
end if
end if
%>
<style type="text/css">

</style></HEAD>

<table width="200" border="0" align="left" cellpadding="0"
cellspacing="0">
<tr>
<td rowspan="3"><img src="images2/Header_01.gif" width="72"
height="178"></td>
<td rowspan="3"><img src="images2/Header_02.gif" width="81"
height="178"></td>
<td rowspan="3"><img src="images2/Header_03.gif" width="194"
height="178"></td>
<td><img src="images2/Header2_04a.gif" width="200"
height="120"></td>
<td><img src="images2/Header2_05a.gif" width="94"
height="120"></td>
<td rowspan="3"><img src="images2/Header_05.gif" width="130"
height="178"></td>
</tr>
<tr>
<td colspan="2"><p align="center"><A href="start_full.asp"> <br>
<span class="style4">Search Home</span></A></p></td>
</tr>
<tr>
<td height="19"> </td>
<td></td>
</tr>
</table>
<p> </p>
<p>  </p>
<p> </p>
<p> </p>
<p> </p>
<table width="732" border="0">
<tr>
<td width="26"> </td>
<td width="696"><span class="style1">
<%
newquery = true

if NewQuery then
if UseSessions then
set Session("Query") = nothing
set Session("Recordset") = nothing
end if
NextRecordNumber = 1

' Remove any leading and ending quotes from SearchString

SrchStrLen = len(SearchString)

if left(SearchString, 1) = chr(34) then
SrchStrLen = SrchStrLen-1
SearchString = right(SearchString, SrchStrLen)
end if

if right(SearchString, 1) = chr(34) then
SrchStrLen = SrchStrLen-1
SearchString = left(SearchString, SrchStrLen)
end if

if Advanced<> "on" then
CompSearch = "{freetext} " & SearchString & "{/freetext}"
else
CompSearch = SearchString
end if

if all<> "on" then
CompSearch= "#filename *.pdf" & SearchString & "{/freetext}"
else
CompSearch=SearchString
end if

if Decs<> "on" then
CompSearch= "#filename DEC" & SearchString & "*.pdf"
else
CompSearch=SearchString
end if

set Q = Server.CreateObject("ixsso.Query")
set Util = Server.CreateObject("ixsso.Util")
'Q.allowEnumeration =True

Q.Catalog = "OfficialDocs"
Q.Query = xxx
Q.allowEnumeration =True

'RESPONSE.WRITE (xxx)
Q.SortBy = "write[d]"
Q.Columns = "DocTitle, vpath, filename, size, write,
characterization, rank"

if MaxResults <> -1 then
Q.MaxRecords = MaxResults
end if
IF 1=1 tHEN
'if FormScope <> "\" then
'Util.AddScopeToQuery Q, "/" ,"deep"
'end if
END IF
if SiteLocale<>"" then
Q.LocaleID = Util.ISOToLocaleID(SiteLocale)
end if

set RS = Q.CreateRecordSet("nonsequential")
'Response.Write (RS.RecordCount)

RS.PageSize = PageSize
RS.CacheSize = PageSize
ActiveQuery = TRUE

'elseif UseSavedQuery then
' if IsObject( Session("Query") ) And IsObject( Session("RecordSet")
) then
'set Q = Session("Query")
'set RS = Session("RecordSet")

'ActiveQuery = TRUE
'else
' Response.Write "ERROR - No saved query"
'end if
end if

if ActiveQuery then
if RS.RecordCount <> -1 and NextPageNumber <> -1 then
RS.AbsolutePage = NextPageNumber
NextRecordNumber = RS.AbsolutePosition
end if

if not RS.EOF then
%>
<br>
<%
LastRecordOnPage = NextRecordNumber + RS.PageSize - 1
CurrentPage = RS.AbsolutePage
if RS.RecordCount <> -1 AND RS.RecordCount < LastRecordOnPage
then
LastRecordOnPage = RS.RecordCount
end if

Response.Write "Documents " & NextRecordNumber & " to " &
LastRecordOnPage
if RS.RecordCount <> -1 then
Response.Write " of " & RS.RecordCount
end if
Response.Write " matching the query " & chr(34) & "<I>"
Response.Write Server.HTMLEncode( term ) & "</I>" & chr(34) &
".<P>"
%>
<% if Not RS.EOF and NextRecordNumber <= LastRecordOnPage then%>
</span></td>
</tr>
</table>
<br>
<table border=0>
<colgroup width=105>
<% end if %>

<% Do While Not RS.EOF and NextRecordNumber <= LastRecordOnPage

' This is the detail portion for Title, Abstract, URL, Size, and
' Modification Date.

' If there is a title, display it, otherwise display the filename.
%>
<p>
<%
' Graphically indicate rank of document with list of stars (*'s).

if NextRecordNumber = 1 then
RankBase=RS("rank")
end if

if RankBase>1000 then
RankBase=1000
elseif RankBase<1 then
RankBase=1
end if

NormRank = RS("rank")/RankBase

if NormRank > 0.80 then
stars = "rankbtn5.gif"
elseif NormRank > 0.60 then
stars = "rankbtn4.gif"
elseif NormRank > 0.40 then
stars = "rankbtn3.gif"
elseif NormRank >.20 then
stars = "rankbtn2.gif"
else
stars = "rankbtn1.gif"
end if
%>

<tr class="RecordTitle">
<td align="right" valign=top class="RecordTitle">
<%= NextRecordNumber%>.
</td>
<td><b class="RecordTitle">
<%if VarType(RS("DocTitle")) = 1 or RS("DocTitle") = ""
then%>
<a href="<%=RS("vpath")%><%=SearchString%>"
class="RecordTitle"><%= Server.HTMLEncode( RS("filename") )%></a>
<%else%>
<a href="<%=RS("vpath")%><%=SearchString%>"
class="RecordTitle"><%= Server.HTMLEncode(RS("DocTitle"))%></a>
<%end if%>
</b></td>
</tr>

<tr>
<td valign=top align=left>

<br>
<%
' Construct the URL for hit highlighting
WebHitsQuery = "CiWebHitsFile=" & Server.URLEncode( RS("vpath")
)
WebHitsQuery = WebHitsQuery & "&CiRestriction=" &
Server.URLEncode( Q.Query )
WebHitsQuery = WebHitsQuery & "&CiUserParam3=" & QueryForm
WebHitsQuery = WebHitsQuery & "&CiLocale=" &
Util.LocaleIDToISO( Q.LocaleID )
%>

<td valign=top>
<%if VarType(RS("characterization")) = 8 and
RS("characterization") <> "" then%>


<%end if%>
<p>
<i class="RecordStats"><a
href="<%=RS("vpath")%><%=SearchString%>" class="RecordStats"
style="color:blue;">http://<%=Request("server_name")%><%=RS("vpath")%></a>
<br><%if RS("size") = "" then%>(size and time
unknown)<%else%>size <%=RS("size")%> bytes - <%=RS("write")%> GMT<%end
if%></i>
</td>
</tr>
<tr>
</tr>
<%
RS.MoveNext
NextRecordNumber = NextRecordNumber+1
Loop
%>

</table>
<P><BR>

<%
else ' NOT RS.EOF
if NextRecordNumber = 1 then
Response.Write "No documents matched the query<P>"
else
Response.Write "No more documents in the query<P>"
end if

end if ' NOT RS.EOF

if NOT Q.OutOfDate then
' If the index is current, display the fact %>
<P>
<I><B>The index is up to date.</B></I><BR>
<%end if

if Q.QueryIncomplete then
' If the query was not executed because it needed to enumerate to
' resolve the query instead of using the index, but AllowEnumeration
' was FALSE, let the user know
%>
<P>
<I><B>The query is too expensive to complete.</B></I><BR>
<%end if

if Q.QueryTimedOut then
' If the query took too long to execute (for example, if too much
work
' was required to resolve the query), let the user know %>
<P>
<I><B>The query took too long to complete.</B></I><BR>
<%end if%>

<TABLE>

<%
' This is the "previous" button.
' This retrieves the previous page of documents for the query.

SaveQuery = FALSE
if CurrentPage > 1 and RS.RecordCount <> -1 then %>
<td align=left>
<form action="<%=QueryForm%>" method="get">
<INPUT TYPE="HIDDEN" NAME="xxx" VALUE="<%=xxx%>">
<INPUT TYPE="HIDDEN" NAME="Advanced" VALUE="<%=Advanced%>">
<INPUT TYPE="HIDDEN" NAME="sc" VALUE="<%=FormScope%>">
<INPUT TYPE="HIDDEN" NAME="pg" VALUE="<%=CurrentPage-1%>" >
<INPUT TYPE="HIDDEN" NAME="RankBase" VALUE="<%=RankBase%>">
<input type="submit" value="Previous <%=RS.PageSize%>
documents">
</form>
</td>
<%SaveQuery = UseSessions%>
<%end if%>

<%
' This is the "next" button for unsorted queries.
' This retrieves the next page of documents for the query.

if Not RS.EOF then%>
<td align=right>
<form action="<%=QueryForm%>" method="get">
<INPUT TYPE="HIDDEN" NAME="xxx" VALUE="<%=xxx%>">
<INPUT TYPE="HIDDEN" NAME="Advanced" VALUE="<%=Advanced%>">
<INPUT TYPE="HIDDEN" NAME="sc" VALUE="<%=FormScope%>">
<INPUT TYPE="HIDDEN" NAME="pg" VALUE="<%=CurrentPage+1%>">
<INPUT TYPE="HIDDEN" NAME="RankBase" VALUE="<%=RankBase%>">
<%
NextString = "Next "
if RS.RecordCount <> -1 then
NextSet = (RS.RecordCount - NextRecordNumber) + 1
if NextSet > RS.PageSize then
NextSet = RS.PageSize
end if
NextString = NextString & NextSet & " documents"
else
NextString = NextString & " page of documents"
end if
%>
<input type="submit" value="<%=NextString%>">
</form>
</td>
<%SaveQuery = UseSessions%>
<%end if%>

</TABLE>

<% ' Display the page number

if RS.PageCount <> 0 then
Response.Write "Page " & CurrentPage
if RS.PageCount <> -1 then
Response.Write " of " & RS.PageCount
end if
end if
%>
<%
' If either of the previous or back buttons were displayed, save
the query
' and the recordset in session variables.
if SaveQuery then
set Session("Query") = Q
set Session("RecordSet") = RS
else
RS.close
Set RS = Nothing
Set Q = Nothing
set Util = nothing
if UseSessions then
set Session("Query") = Nothing
set Session("RecordSet") = Nothing
end if
end if
%>
<% end if %>
<BR>
<TABLE ALIGN="CENTER" WIDTH="100%">
<TR>
<TD ALIGN="CENTER"> <font size="3" face="Verdana, Arial, Helvetica,
sans-serif">
<b><A href="start_full.asp"> Search Home </A> </b>
</td>
</tr>
</table>


</BODY>

<SCRIPT LANGUAGE=JSCRIPT>
function Init()
{
QueryForm.SearchString.focus();
}

function msieversion()
{
var ua = window.navigator.userAgent;
var msie = ua.indexOf("MSIE");
if (msie>0)
r
}
</script>

Thanks!

Stu

2005-04-20, 7:48 am

> However, when I JUST want to search for multiple filenames without the
> free text portion of the query, the search pages hangs and eventually
> times out. I've tried the following:
>
> #filename=abc* OR #filename=def*
> (#filename=abc* OR #filename=def*)
> (#filename=abc*) OR (#filename=def*)
>
> I know the file is indexed properly since I can find it like this:
>
> #filename=<filename>

Indexing service does not handle ORs too well. You may well be faster
submitting a search for the files one at a time then joining them up at in
the client/asp.

If you take a step back and ask how you know which files you want returned
you might be able to construct a faster query based on the document content.

Cheers
Stu