NMPDR Development Blog

A few fixes to the lucene search have been posted to the Development NMPDR.

• The search results page knows whether results were returned or the advanced form was displayed, and this information is used by the template to insure the proper help text is displayed.
• The organism names now display correctly with the strain included. Previously, the NMPDR group name was showing instead of the strain.

RNAs once again work correctly on the NMPDR Development version.

To test the fix,

1. Go to the Development Server home page.
2. Enter fig|100226.1.rna.10 into the search box.
3. Select the NMPDR button on the results page to view the feature's data.

Subsystem diagrams now display as NMPDR pages when invoked from Sprout. To test this fix, go to this subsystem page, then click on the diagram link.

Subsystems displayed in the signature genes tool now contain a link to the appropriate subsystem page. To test this, go here, then

• Select Campylobacter jejuni RM1221 as the given gene.
• Select Campylobacter jejuni RM1221, Campylobacter jejuni subsp. jejuni 260.94, Campylobacter jejuni subsp. jejuni 81-176, and Campylobacter jejuni subsp. jejuni 84-25 for set 1.
• Select Campylobacter jejuni subsp. jejuni CF93-6, Campylobacter jejuni subsp. jejuni HB93-13, and Campylobacter jejuni subsp. jejuni NCTC 11168 for set 2.

Click Find Discriminating Proteins. You should see at least three linked subsystems in the discriminating set.

The essentiality page works again. You can find it here.

Last Monday, after a phone conference, I removed the property and feature type controls from the feature filter in the new search system in order to insure we only returned meaningful results and to reduce the problem that most property searches return no results. This greatly streamlined the code, but it broke the essentiality page, which was using the feature filter to search for genes with specific properties. To fix the problem, I added a new search that looks for the occurrence of specific property name/value pairs in a chosen genome. This is not a very useful search for users visiting the site, but it does make it possible to do the essentiality searches using the old essentiality properties. I am now leaning toward the idea that the generated search page (found here on the development server) will only be used by developers, and we can have another page for the user-friendly searches that can have more involved explanations and examples.

Anyway, the important thing is that essentiality is back.

The organism page search boxes have been fixed. To test this, go to the main page on the Development Server and type 2.7.6.3 into the search box. Several hundred results will come back. Next, go to the campylobacter page and try the same search. You will get a much smaller set of results, and all of them will be for campylobacters.

Hopefully this will be the last Lucene fix and we'll be using the new search in version 16.

I have changed the delimiter for subsystem classifications from space to colon. This fixes the problem with the classification names being truncated to one word. In addition, I updated the genome statistics page to show the name as well as the number of the genome.

To test this, go to the subsystem summaries page, select Listeria monocytogenes EGD-e, and click Show Subsystems.

PSI-Blast is sort of working on the NMPDR Development Site.

PSI-Blast was reported non-functional in v14. The reason had to do with the fact that the NCBI tools assume the first form on the web page is the NCBI tool's form, when in fact it was the third form when the results were loaded into an NMPDR template. All of the protein page tools use a single template, so the fix was to change that template to eschew the search and bug reporting forms normally present on every NMPDR page.

Unfortunately, the Sprout database is still loading, so this fix can't be tested directly. Instead, you must use the URL you would get if you clicked on the PSI-Blast link at the bottom of a protein page once the protein page is working. For the infamous fig|83333.1.peg.4, click here to get the desired page. Clicking on the FORMAT button from this location used to take you to a blank search results page; now it takes you to the PSI-Blast results.

My intent is to test this as soon as the load completes and then slipstream the fix into the live site. If the load is still going on tomorrow, then I will slipstream without first testing on the development site, which is scary, but required if we're to be ready for a Wednesday demo.

Two web page changes have been slipstreamed into the live site.

1. The tool template has been changed so that Psi-Blast works correctly on the NMPDR protein page.
2. The essentials page has been fixed so that the two E coli bars work. Previously, these had been left out of the image map and clicking them had no effect.

These changes have also been applied to the development site.

The inclusion criterion for subsystems has been changed. Starting with the next load, only NMPDR subsystems will be included in the Sprout database. Previously, any usable subsystem was included. The total for this load would have been 549 under the old scheme; now it will be 393.

This is actually a reversal of a change made on July 13 of this year. The What Goes Into Sprout article has been updated accordingly.

The subsystem classification problem has been fixed in version 16. To test this, go to the Subsystem Summaries Page, select Vibrio cholerae O1 biovar eltor str. N16961 and click Show Subsystems. Alanine, serine, and glycine will now be a subclass of Amino Acids and Derivatives rather than part of the classification name.

The default keyword search mode has changed from OR to AND. I accomplished this by programming the search to put a plus sign in front of each bare word. The result is that Vibrio 2.7.6.3 will now only return features with EC role 2.7.6.3 in Vibrio organisms. To get features in Vibrio OR with role 2.7.6.3, you put parentheses around the words: (Vibrio) (2.7.6.3).

The search form now appears after the results on the search page. This turned out to be a very easy fix: I just changed the template (SproutSearch_tmpl.php).

The Search button in the Genome Menu has been removed. When doing a string-select search in the genome menu, the selection will take place as soon as the focus moves away from the text box. The instructions have been updated to say that you type in a string and press TAB instead of typing in a string and clicking the button. The search has also been made case-insensitive.

The Advanced button has been sprinkled around the site again. It takes the user to the FidSearch module. An advanced-search button is also now available on the word search page, since that's where a standard search request will land the user.

• The active row subset box on the subsystem display page is now fixed at 5 rows. To test this, go to this page. In the previous version, the box was 108 rows high, which made it difficult to see stuff below it.
• Quoted strings in a search are now treated as required rather than optional. To see this, go to the main page and enter Campylobacter "membrane protein". Previously, this would return everything in Campylobacter, because the membrane protein phrase was considered optional. With the fix in place, only membrane proteins for Campylobacter will b/e returned. To get the old behavior, use parentheses: Campylobacter ("membrane protein").
• The drug targets script has been modified to put the bulk of the explanatory text in the templates so that it can be updated by the content team. The if-then logic present in the original CGI has been converted to TMPL_IF constructs in templates/targets_tmpl.php.
• The heading for the first column in the search results has been changed from Gene Name to Organism and Gene ID

Later tonight I will run through the attribute stuff and reload the property table.

• The special options (sort by role, show subsystems, show aliases) have been added to the keyword search page.
• The special options were missing from the Gene Search and the Signature Genes Tool. This has been fixed.
• The Gene Search returned an error if no search words were specified. This has been fixed.
• The keyword search box has been moved to the bottom of the Subsystem Search Form.
• The presentation order for the controls on the Signature Genes Tool has been changed.
• The Show URL option is now the default on all of the search forms.

• The GO attributes from Ross and Kaitlyn have been loaded into the attribute system on the Development Server and copied into the NMPDR data base. An example GO attribute can be seen on the Streptomyces coelicolor peg 3361 page.
• The feature filter rows have been moved to the bottom of the signature genes form.
• The drug target page has been updated. I attempted to combine independent changes from both Matt and Leslie.

The text index on the Feature table was missing, which was making all the searches very slow. I have rebuilt the indexes on both the Development and Staging servers, and it helps.

• The text search index somehow got dropped, making keyword searches very slow. This has been fixed.
• I have modified the BBH server to accept batch requests. When requesting BBH data, you can specify an SQL pattern instead of a feature ID. So, for example, fig|100226.1.% would compute BBHs for all features of Streptomyces coelicolor. To control the bandwidth in cases like this, you can also specify a list of target genomes. Only BBHs that land in the target genomes will be returned by the server. This dramatically improves the performance of the Signature Genes Tool.
• The mini-forms on the search results page (NMPDR and GBrowse have been converted to fake buttons so that they work after a copy and paste and may be right-clicked to open the results in a new window or tab. They look like real buttons, but do not have an animated click effect.

These fixes, along with a whole bunch of web page improvements by Leslie, have been moved to the staging server.

• The Vibrio group now consists only of pathogenic Vibrio and other Vibrio. Previously, non-pathogenic Vibrio were further specialized, but since there were only two of them, it didn't make a lot of sense.
• The bug in ERDB that prevented the Feature text search index from building has been fixed.
• The search URL is always displayed, and there is no longer a Show URL checkbox.
• If the user presses ENTER on a search form and we are not displaying results, an attempt is made to search. This will not make any difference in Firefox, but it means that IE users do not need to manually click the GO button.

The load files have been created for all the tables except the Feature table. The Feature table load stalled because of the latency required to communicate with the attribute server. I changed it to retrieve attributes once per genome instead of once per feature and it is now moving considerably faster.

• The new Listeria genomes (393133.3, 401650.3, 393128.3, 393117.3, 393121.3, 393123.3, 393124.3, 393125.3, 393130.3, 393131.3) have been marked as NMPDR, so they now appear on the Listeria page. This change will need to be ported to the annotator SEED.
• The Listeria and Strep search formlets had spelling errors in the form details, resulting in a 404 error. These have been fixed.
• There is now an anti-spam CAPTCHA code protecting the mailing list signup page. The mailing list has been hit hard by spammers. Unfortunately, the change required removing the signup formlets on the organism pages.

I have commented out the overflow-x and overflow-y rules in the main NMPDR CSS file. The directional overflow rules are still in the not-quite-supported category of CSS. In the Mac version of Firefox 2, when the content height became too large, the browser went berserk and created a ghastly blank page with things popping in and out of existence. This is, in defiance of all sense and logic, only a problem on the Mac. On Windows the page displayed just fine.

The page that exhibited the problem was the database design page.

Thanks to Ross and Kaitlyn for donating their laptops so I could test this problem.

In other news, there will be a delay in the schedule for implementing attributes and rolling out NMPDR version 18 because I still can't use VPN. Finding a way to work around that will be my main focus tomorrow.

• A previous fix that allowed Internet Explorer users to press ENTER from a search form to start a search was not working properly, with the result that users would get an error message when they were linking directly to the forms. This fix has been backed out.

The festival will continue later, after I fix an emergency problem with attributes on the NMPDR.

• The drug target pages are working again. The drug target database and search capabilities will be put off until version 19.
• The unclassified subsystems have been moved to the bottom of the subsystem tree.
• Links to the attribute server control panel and the documentation directory are now available directly from the sidebar on the development blog.
• The Signature Genes search now displays the match score. In addition, you can ask for the matching genes to be displayed. This will show the genes that were BBHs from the reference gene to the target set genomes.
• The ShowCounts script has been updated to output the genome list control to all_genomes.inc instead of subsystems.inc. The organism summary page was not being updated properly due to the inconsistency of the file names.
• The search word box has been moved to the top on the gene search page.
• If there is only one type of advanced search (which is currently the case), the table of contents is not displayed on the advanced search page.
• The feature loader has been changed so that the functional assignment for non-PEG features are taken from the annotations instead of the alias list.

The organism group pages are now fixed. The problem is that the structure of the Genome records changed, so attempts to compare the new data to the old data got ridiculous results. I believe I know how to overcome this problem, but it will take some work. For now, the important thing is that all of the pages are correct.

The subsystem tree has been changed so that show-clusters is on by default.

More to come later...

I fixed the version-comparison utilities so that the old database uses the old version of the DBD. This fixed the problem with the bogus NEW! markers in the group pages. I am running the main difference report even as we speak, so we can see what's changed on a genome-by-genome basis between version 18 and version 19.

The tentative cutover date is April 6, after Leslie and Claudia's visit.

I've added an animated banner to the intermediate search status page. My experience when I was in industry is that if there's a moving picture involved, it's easier to believe the search is working. I have also disabled the Annotation button so it won't show up in the final result.

The difference report is now available. It has also been put in the Quick Links list on Development Blog. The main anomaly is that none of the 29 new genomes have BBHs.

I am currently working on a glitch in the subsystem tree and a method for downloading search results. (This latter I hope will help us to understand the peculiarities of the signature genes tool.)

The annotation history links have been missing from the NMPDR Protein Page for some time. This is now fixed.

We now have the ability to download search results to a tab-delimited file. On each search results page (for example, this one), there is now a link titled Click here to download the full search results. Clicking the link will open a download-file box and then download the search results to a specified location on your workstation. The downloaded text is cleaned of links and does not include the button columns. There is one anomaly. In Firefox, the OK button in the download dialog box appears disabled, but will work properly when clicked. (Even Tobi does not know how to fix this problem, so I don't expect an answer soon.) In the meantime, this is a useful feature I have wanted for some time and is worth keeping.

• The search box on the strep page has been fixed. Previously, it gave an error message any time you tried to search.
• The Compare Regions button on the protein page now returns the correct results. Previously, it was using BBHs instead of Sims to detect the closest genes.
• The category column in the organism drug target tables was always coming up with the Suggest link. This has been fixed. In addition, the Suggest link has been turned off so that the categories show up better.
• The last of the damaged image files have been repaired.

I am still working on the problem Ross found with the NMPDR PINS display not matching the SEED display. I know which subroutine in the Chromosomal Clusters CGI is failing, but I don't yet know where or why.

• The download link on the search results page is now marked as application/octet-stream instead of text/plain. This means it is more well-behaved in the browser, but it also means the line-endings have to be set to the proper value by the search CGI script rather than by the browser. The web page uses Javascript to detect the client operating system and then passes that information to the CGI script, which puts the correct line-endings in the file.

Remaining tasks include finding the problem in the Sprout version of the Pins (I now know it has something to do with setting the colors) and adding the serotypes to the NMPDR organism group pages.

• The Pin buttons are finally working again on the NMPDR development server protein page.
• The serotypes are now permanent fixtures on the core organism group pages (e.g. Listeria). The serotype data that Leslie entered manually some time back has been stored on the attribute server. The page generator looks at all the genomes in the group, and if at least one has a known serotype, the serotype column appears to the right of the genome name.

• The Listeria innocua and Listeria welshmeri have been reclassified as belonging to the other Listeria group. In addition, their serotype information has been loaded onto the attribute server and now appears on the Listeria group page.
• All the numerous changes over the past month have been checked into CVS, including the massive renaming of DBObject to ERDBObject so that it no longer conflicts with PPO, the new Sprout methods for SEED Viewer support, the fix to Pins and Compare Regions so that they work properly in Sprout, and a bunch of other small stuff.

The Development NMPDR is now code complete for version 19 (which is due to move to the staging server on Wednesday night). This means all we have left are bug fixes and cover page improvements.

The PEGs Only check box has been removed from the signature genes tool. In almost all cases it makes no difference. When that situation changes, we can add it back.

• The subsystem tree was displaying incorrectly in Internet Explorer. This has been fixed.
• It is now possible to download search results in FASTA format.
• You can now specify a favored alias type (e.g. uni, gi) for searches. The favored alias type is shown at the beginning of the alias list and is also used as the gene ID in the FASTA downloads.

These changes have been copied to the Staging Server,

• The subsystem count at the bottom of the organism summary page is no longer be active as a link. Previously, this took the user into the SEED and there was no real way to get back from that. It was easier to simply disable the link in NMPDR than convert the entire subsys_vector script.
• The downloads page presents an FTP directory when the user clicks the FTP link. When the user clicks the HTTP link, a ZIP file of the directory is downloaded to the user's hard drive.
• There is now an Operon search. Like the drug search, it's available on an orphaned page not generally visible to the public. The user specifies a genome and some tuning parameters and is presented with the genes in the genome grouped by operon. In addition to the standard search results fields, the operon search also displays the feature location and the feature's upstream DNA. For forward operons, the genes are presented in the normal order. For backward operons, the genes are presented in the reverse order so that the upstream data for the first gene of an operon is always upstream of the entire operon. This is almost what Ron Taylor wanted, the exception being that the downloaded file only includes the subsystem names and not the links. My goal here was to prove I could create a custom search in one day. It was one-and-a-half days, but I'm getting closer every time.

All of this has been moved to the staging server, and the links herein are staging server links.

• There is now a FIG method (get_group_key_info) that returns the names and descriptions of the attribute keys in a group.
• Updated the commentary page for chromosomal clusters to get the evidence codes in a single batch rather than one at a time.
• Fixed the homolog clusters page to use an NMPDR template.
• Created a search for all the docking results for a PDB. This is the first search that returns objects other than features, which is an important advance.

There are still some things to be fixed, but I'm working on it.

• The output from the BLAST search is now sorted correctly. Some genes weren't being correctly identified as NMPDR core genes, causing them to float to the bottom of the list.
• The context button on the BLAST/Scan result page has been moved after the NMPDR button.
• The old favored-alias feature has been restored. In addition, when the selected ID type is not FIG, the FIG ID is shown in the organism name column in the familiar old square brackets.
• The formlets on the search output have been changed to fake buttons. This allows users to right-click on the buttons and open the linked pages in another tab or window.

I believe this covers the NMPDR-specific issues Leslie had with version 21. The remaining issues deal with the protein page and will be addressed separately.