NMPDR Development Blog

There are apparently several different ways to make data downloadable and transportable to Excel. I believe the miscommunications between me, the users, and the coders stems from using different combinations of browser/os and also from different expectations. Providing a link on an html page that takes text out of an html table and puts it on the screen so that it looks like a tab-delimited text file is what is frequently provided as a service called "download tab-delimited text" or "export file for Excel".

In order to actually put this html into Excel from Firefox on a PC requires copying the text on the screen, opening Excel, and choosing "paste special" from the edit menu. Then change from html to text and paste. It doesn't work if you just paste. Thus, there is no reason to redraw the web page because one can select and copy any html table and paste it into Excel the same way. Drawing an ugly html page does not facilitate viewing the info in Excel.

Most biologists will also interpret "download" or "export" as meaning a file will appear on the local computer; not that an ugly web page will appear and do nothing--not even prompt you to copy it. I can provide a help statement that can go on the template for this type of "exported" file that will explain what to do with it in the common browser/os combinations. This will help a lot.

PC users who run Internet Explorer have the easiest time--any html table that is copied can be pasted directly into Excel, retaining the links. This is ultimately what I want any user to be able to do--transport the search results or the protein context table or the BBH into a table in Excel WITH the links to NMPDR.

I have completed proof-of-concept testing for keyword searching in Sprout. Each feature will have a keyword list in the database that is processed by MySQL into a text-search index. In addition to simply listing keywords, it is possible to put modifiers on the words. For example, dnaK -hypothetical would return all dnaK features which are not hypothetical. A complete description of the operators is available here, but the point is we are already ahead of what Lucene can deliver, and it's better controlled. For example, if you ask for all hypothetical features for NMPDR genomes that belong to a specific subsystem, the search tool will combine the keyword search with the other criteria so that we get the full benefit of the database indexing.

The keyword search is currently inoperative while I load the keywords into the feature table. However, once its ready it will be automatically incorporated into all search tools that support feature filtering. There will also be a special keyword-only search tool designed to replace the ubiquitous NMPDR search box.

Keyword searching is now available on the unpublished Sprout Search Page. Not all the keywords we want are working yet, but you can still experiment with things like EC numbers and words found in functional assignments.

The feature filter has also been redone to make it less complicated. The only filters remaining are the subsystem name and the keyword search box. This change was made because too many of the other filtering choices (especially properties) tended to produce improper or confusing results.

The new search has been installed on the Development Site. The old nmpdr_lucene_search.cgi script now redirects the request to the new search module so that the Mozilla search bar will continue to work. I've tested that as best I can without modifying the live site.

I have filled in the various help templates (all called SearchHelpsomething.inc in the template directory), but I suspect they'll need additional work. I'm also thinking of delaying the rollover while we work out the bugs.

There is now a subsystem search page on the NMPDR Development Site. It contains a collapsable tree of the subsystems, and you can choose any node in the tree to get all the features in the subsystems below it. In addition, each leaf node links to the NMPDR display page for the subsystem.

I'm not entirely comfortable with this page, because it feels like I should be able to click somewhere and find out what the subsystems do or what the terms mean. Unfortunately, that information is not in the database, and I'm not qualified to make it up on my own.

In any case, we finally have a directory of subsystems on the NMPDR, which has been a glaring omission for some time now, and this has enabled me to move in Leslie's new search instructions page, which will hopefully make the site easier to use.

Several changes to the new search facility are now available on the Development Site.

• Search parameters are now remembered during page navigation. Previously, if you navigated to a different page in the search results, the search form at the bottom of the page would be reset. Now it remembers, allowing you to refine your original search.
• Blastx has been added to the blast tools page and the form has been redesigned to make it easier to use. Finally, the activation button caption has been changed from Go to Blast.
• There is a table of contents on the Advanced Search Page so that users can get to the search they want more quickly.
• The NMPDR and GBrowse buttons in the search results are now real buttons instead of hyperlinked images. In addition, the styles have been updated in NMPDR.css to fix the glitch that caused the button text to shrink when you clicked it.
• The word feature has been replaced with gene in all the explanation, caption, and help text. The link to the FidSearch tool on the Search page has been changed from Advanced to Genes to distinguish it from the Advanced button.
• The search box on the genome list control is once more button-activated. The button contains the text describing the box's purpose.

The NMPDR cover pages have been moved to nmpdr-1 from web-1. Combining the servers makes all sorts of things easier and faster. The new development server address is http://nmpdr-1.nmpdr.org/next/.

The latest difference report is here. The genome-related data is now in a separate table. The old report listed all the new and deleted features, but those lists are now so large that they make the report difficult to read. Instead, the number of new and deleted features will be added to the genome report (starting with the next one). The new report contains all the data items for the semi-annual report, which means that from now on there will be no need to write custom applications every time the report comes around.

A link to the report has been placed on the front page of the NMPDR Development Blog under the title Latest Version Database Report so that it can be found easily when it is needed.

The NMPDR Viewer now allows downloading each of the three result trees from a metabolic reconstruction. The downloads are done direct-to-disk, like the downloads from the NMPDR search results page.

This was implemented as a response to a customer requirement for the RAST Server. The next step is to move the code over to the RAST server itself.

The context view available from the BLAST/Scan search now includes additional explanatory text at the top. The pop-up box you get from clicking on a gene now shows only the NMPDR protein page. I am awaiting a response from Toby and Daniel as to whether I can get the click to go directly to the page without the intervening popup box.

When the incoming search pattern or sequence is more than 30 characters, the context view displays a character count rather than the entire sequence, because a long sequence looks really ugly.

I am in the process of re-doing the search helps and converting them to pages in the Seed Wiki. Clicking on the new-fangled blue-question-mark-things will pop up the relevant Wiki page.

In addition to the Wiki conversion, I have 11 more change requests from Leslie and 9 requests from Folker and Liz. Given that the Wiki conversion is essential and I also have to update some of the cover pages myself due to Leslie's absence, I am seriously thinking of pushing ahead the release date by another week or two. I will decide this tomorrow morning based on how much pain I'm in when I wake up.