NMPDR Development Blog

Bruce went to see the dentist today and he is feeling much better; however, he was unable to return to work due to a mishap involving a tube of super-glue and a broken car mirror.

Respectfully submitted,

Ferdinand T. Cat

Today, Bob and I worked out the architecture for the new attribute server. The server itself will be on the protected anno-3 machine, and the control panel that allows you to define and delete attribute keys will be available only on that machine. Access to attributes (which includes inserting, deleting, and retrieving) will be performed using XML RPC. XML RPC allows a PERL object to appear as a normal object even though it is running on a remote server. Initially, the RPC interface will be providing just the minimal set of methods necessary to implement attributes; but this same scheme with a larger set of methods could allow us to put multiple full-function ERDB databases on remote servers with very little coding effort.

Initially, the new attribute methods will be implemented as replacements for the current methods in FIG.pm. There will be two diminutions of functionality: we can no longer search on URLs, and there is no provision for controlled-vocabulary attributes. Hopefully, this will not be a big problem.

A list of the methods being implemented is given below the fold.

Keyword searching in Sprout is implemented using the text search capabilities of MySQL. Each feature has a keywords field that contains a space-delimited list of all the keywords for that feature. The keyword list contains:

• The FIG feature ID and all aliases.
• The functional assignment, if any.
• The names and classifications of the subsystems containing the feature.
• For each subsystem role performed by the feature, the role name and abbreviation.
• The genome ID and the complete taxonomy (including the names of the genus, species, and strain).
• The special attribute names relevant to the feature (currently essential, virulent, and/or iedb).

Words containing hyphens or underscores are included in their natural form as well as being broken up into pieces. There is also some folding done. All letters are changed to lower-case, hyphens (-) and periods (.) are converted to underscores, and vertical bars (| and colons (:) are converted to apostrophes ('). This same process occurs on the keyword phrase before it is applied to the keyword index. There are technical requirements for this (apostrophes and underscores are the only punctuation marks allowed by MySQL in searchable text), but it also makes the search process more forgiving.

Yesterday Matt, Jen, and I put up some new drug target pages on the Development Web Site. The entry point for the new pages is http://web-1.nmpdr.org/next/content/docking_proof_of_principle.php. The entry page shows the three PDBs used as examples in Gohar's thesis, and contains links to some charts and to a page that displays the ligands and the docking results.

The ligands page is generated from an updated version of the ligands.cgi script from the bio servers. The major change from the original script is that we show 50 ligands at a time instead of all 10,000 on a single page. (The latter ran into problems due to the template philosophy used elsewhere in Sprout. So that the best results were shown first, we used a utility to sort the data files so the examples with the lowest binding energy appear first.

Hopefully, this will be the first of many pages of drug target results we can show on the NMPDR.

A drug target search is now available on the NMPDR. This new search allows you to select one of the drug trarget projects from a dropdown menu and display all the features found by that project. The search styles have been changed to make the results tables use the alternating-color scheme used elsewhere on the site.

The drug target search page can be found at http://web-1.nmpdr.org/next/FIG/SearchSkeleton.cgi?Class=DrugSearch.

Please let me know if you feel we need more or different columns, or even additional searches for drug target data.

• New Development Site loaded: October 31, 2006
• Latest code and data copied to Staging Site for beta testing: November 13, 2006
• Version 16 published to Public Site: November 18, 2006

Part of the delay is because I had to spend some time on the new attribute system, and the other part is that one of the U of I classes has an assignment due Nov 15, and their instructions are relevant to the version 15 setup.

The default keyword search mode has changed from OR to AND. I accomplished this by programming the search to put a plus sign in front of each bare word. The result is that Vibrio 2.7.6.3 will now only return features with EC role 2.7.6.3 in Vibrio organisms. To get features in Vibrio OR with role 2.7.6.3, you put parentheses around the words: (Vibrio) (2.7.6.3).

The search form now appears after the results on the search page. This turned out to be a very easy fix: I just changed the template (SproutSearch_tmpl.php).

The Search button in the Genome Menu has been removed. When doing a string-select search in the genome menu, the selection will take place as soon as the focus moves away from the text box. The instructions have been updated to say that you type in a string and press TAB instead of typing in a string and clicking the button. The search has also been made case-insensitive.

The Advanced button has been sprinkled around the site again. It takes the user to the FidSearch module. An advanced-search button is also now available on the word search page, since that's where a standard search request will land the user.

The new attribute system is now in place. The four major attribute methods (get_attributes, add_attribute, delete_attribute, and change_attribute) in FIG.pm have been modified so they will process attributes according to how the system is configured.

1. If FIG_Config.pm contains a value for $attrURL, then FIG.pm will look for the attributes at the specified URL. Currently, a set of live attributes are located at http://anno-3.nmpdr.org/attrib_server/AttribXMLRPC.cgi. (That address is an XML service script, not a real web page.)
2. If FIG_Config.pm contains a value for $attrDBD, then the FIG.pm will look for the attributes from a MySQL database. Currently, there are MySQL databases for this purpose set up on anno-3 and nmpdr-1.
3. If neither of those values are found in FIG_Config.pm, the legacy attribute system will be used.

Eventually, we want a local attribute database (method 2) set up on the bioseed servers for use as a scratch database, and public SEEDs would use the attribute server (method 1). There is a timing issue with the conversion, and the messy details are given below the fold.

There is now a subsystem search page on the NMPDR Development Site. It contains a collapsable tree of the subsystems, and you can choose any node in the tree to get all the features in the subsystems below it. In addition, each leaf node links to the NMPDR display page for the subsystem.

I'm not entirely comfortable with this page, because it feels like I should be able to click somewhere and find out what the subsystems do or what the terms mean. Unfortunately, that information is not in the database, and I'm not qualified to make it up on my own.

In any case, we finally have a directory of subsystems on the NMPDR, which has been a glaring omission for some time now, and this has enabled me to move in Leslie's new search instructions page, which will hopefully make the site easier to use.

The NMPDR will be using a new approach for the drug target project beginning in version 17.

Instead of working tables, the drug target data will be divided into three categories: Drug, Vaccine, and Toxin. The data will be curated manually before it is presented to the NMPDR to insure that the PDB associated with a PEG is a qualified match.

Version 17 will start loading Friday evening the 17th. At that time, three flat files (one per category) need to be available at /disks/nmpdr/v16/FIGdisk/FIG/var/DrugTargets so they can participate in the load.

A new script has been written on the bioseed server that allows users to focus on data for a particular organism or a particular category. This script will be adapted for NMPDR and then linked into the drug target pages. The new script will go live on December 1.

The attribute server is now live. In addition, there is a new script-- ExportAttributes-- for writing attributes to flat files.

To use the attribute server, simply put

$attrURL = "http://anno-3.nmpdr.org/attrib_server/AttribXMLRPC.cgi"

in your FIG_Config.pm file. This server should not be used for testing code that adds and deletes attributes. To do that, use

$attrURL = "http://nmpdr-1.nmpdr.org/next/FIG/AttribXMLRPC.cgi"

Attribute keys are now a controlled vocabulary, and you can only attach attributes to genomes and features that are known to the server. If you need to create a new attribute key, you must use the Attribute Maintenance Page. (For the testing server, use this page instead.)

Currently, only Genomes and Features can have attributes, but the possibility exists of adding attributes to subsystems and other objects if needed.

I reloaded the NMPDR 16 database. The new difference report is below the fold.

Several changes to the new search facility are now available on the Development Site.

• Search parameters are now remembered during page navigation. Previously, if you navigated to a different page in the search results, the search form at the bottom of the page would be reset. Now it remembers, allowing you to refine your original search.
• Blastx has been added to the blast tools page and the form has been redesigned to make it easier to use. Finally, the activation button caption has been changed from Go to Blast.
• There is a table of contents on the Advanced Search Page so that users can get to the search they want more quickly.
• The NMPDR and GBrowse buttons in the search results are now real buttons instead of hyperlinked images. In addition, the styles have been updated in NMPDR.css to fix the glitch that caused the button text to shrink when you clicked it.
• The word feature has been replaced with gene in all the explanation, caption, and help text. The link to the FidSearch tool on the Search page has been changed from Advanced to Genes to distinguish it from the Advanced button.
• The search box on the genome list control is once more button-activated. The button contains the text describing the box's purpose.

After much delay, heartache, and gnashing of teeth, version 16 of the NMPDR has been moved to the Staging Site. Please give it a once-over to make sure your favorite features still work. If there are no problems, it will go live some time on Thursday.

The new drug target data is not present in this version. It will be added for the next version, which is scheduled to go live on December 4.

• New Development Site loaded: November 27, 2006
• Latest code and data copied to Staging Site for beta testing: November 30, 2006
• Version 17 published to Public Site: December 4, 2006

Version 17 of the NMPDR is now available on the development server. It includes the new drug targets pages, which can be seen at http://web-1.nmpdr.org/next/FIG/targets.cgi, though the information there has not yet been completely curated.

Not all of the attributes we want are available. Once the attribute system is fixed (hopefully in a day or two), I will reload the NMPDR property table.

I ran into a problem loading the attributes, so the new attribute system is still offline; however, I have a list of the keys and my attempt at a description of what each one means. They are arranged in a table below the fold.

Please let me know if I am describing any of the attributes incorrectly.

• The active row subset box on the subsystem display page is now fixed at 5 rows. To test this, go to this page. In the previous version, the box was 108 rows high, which made it difficult to see stuff below it.
• Quoted strings in a search are now treated as required rather than optional. To see this, go to the main page and enter Campylobacter "membrane protein". Previously, this would return everything in Campylobacter, because the membrane protein phrase was considered optional. With the fix in place, only membrane proteins for Campylobacter will b/e returned. To get the old behavior, use parentheses: Campylobacter ("membrane protein").
• The drug targets script has been modified to put the bulk of the explanatory text in the templates so that it can be updated by the content team. The if-then logic present in the original CGI has been converted to TMPL_IF constructs in templates/targets_tmpl.php.
• The heading for the first column in the search results has been changed from Gene Name to Organism and Gene ID

Later tonight I will run through the attribute stuff and reload the property table.