The Database Interface Classes¶
For each API endpoint exposed in the Django app, there is a corresponding class that provide methods to execute CRUD operations asynchronously.
There are two types of API endpoints – those that contain only records data, and those that store both records and pointers to files.
Connecting to the Database¶
The database currently runs on HTCF service partition. This is a single node with 8 CPU and 30 GB that is meant for long running low resource jobs. The components that need to run are a postgres database, a redis instance and the django app. As long as these components are running on the service partition, you can connect via an ssh tunnel with:
ssh username@login.htcf.wustl.edu -N -L 8001:n240:8000
where the 8001:n240:8000 takes the form of local_port:cluster_node:app_port. The
django app will always be served on port 8000, and n240 is the only service
partition node. You may choose a different local port.
If you do this and cannot connect, let me know and I’ll check the status of the jobs on the cluster.
Database username and password¶
Once you have a tunnel, you can access the database frontend at 127.0.0.1:8001
(or a different local port, if you changed that number). If you haven’t already
signed up, you’ll need to click the ‘sign up’ button and follow
the instructions. The e-mail server is not hooked up at the moment, so when it says
“see the e-mail”, send a slack message and let me know. I’ll give you a link to
complete the sign up process. After that, you can just use the “sign in” button.
For computational tasks, including generating rank response data, celery workers must be launched on the HTCF general partition. There is currently a script that is meant to monitor the redis queue and launch/kill these workers automatically, but this functionality is new and largely untested. You can monitor the workers/tasks if you create another tunnel with:
ssh username@login.htcf.wustl.edu -N -L 8002:n240:5555
You’d access this dashboard at 127.0.0.1:5555
The login is currently:
username: "flower"
password: "daisy"
(yes, really – the security comes from the fact that you need to login with HTCF)
Configuring the Database Interface Classes¶
The database classes expect the following environmental variables to be set.
BASE_URL='http://127.0.0.1:8001'
TOKEN='<your token="">'
BINDING_URL='http://127.0.0.1:8001/api/binding'
BINDINGMANUALQC_URL='http://127.0.0.1:8001/api/bindingmanualqc'
CALLINGCARDSBACKGROUND_URL='http://127.0.0.1:8001/api/callingcardsbackground'
DATASOURCE_URL='http://127.0.0.1:8001/api/datasource'
EXPRESSION_URL='http://127.0.0.1:8001/api/expression'
EXPRESSIONMANUALQC_URL='http://127.0.0.1:8001/api/expressionmanualqc'
FILEFORMAT_URL='http://127.0.0.1:8001/api/fileformat'
GENOMICFEATURE_URL='http://127.0.0.1:8001/api/genomicfeature'
PROMOTERSET_URL='http://127.0.0.1:8001/api/promoterset'
PROMOTERSETSIG_URL='http://127.0.0.1:8001/api/promotersetsig'
REGULATOR_URL='http://127.0.0.1:8001/api/regulator'
This can be achieved in the package during development with a .env file at the
top most level of the package. The .env file is loaded in the package __init__.py.
If you are importing yeastdnnexplorer into a different environment, then you’ll
need to add the package dotenv and execute load_dotenv(dotenv_path=env_path). If
the .env file is in the same PWD in which you execute that command, there is no
need to specify a path.
Token Authentication¶
Once you have a username and password to the database, you can retrieve your token.
Make sure that you put this token, at least, in a .env file, and make sure that
.env file is in your .gitignore.
Alternatively, you could retrieve and store in memory the token at the beginning of
each session – this is more secure if you are not using a .env file.
The .env file is already in the yeastddnexplorer .gitignore
curl -X 'POST' \
'http://127.0.0.1:8001/auth-token/' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"username": "username",
"password": "password"
}'
Or with python:
import requests
url = "http://127.0.0.1:8001/auth-token/"
headers = {
"accept": "application/json",
"Content-Type": "application/json",
}
data = {
"username": "username",
"password": "password",
}
response = requests.post(url, json=data, headers=headers)
print(response.text)
Using the Interface Classes¶
from yeastdnnexplorer.interface import *
import matplotlib.pyplot as plt
Records Only Endpoints¶
The records only endpoints are:
-
BindingManualQC
-
DataSource
-
ExpressionManualQC
-
FileFormat
-
GenomicFeature
-
PromoterSetSig
-
Regulator
When the read() method is called on the corresponding API classes, a dataframe will
be returned in the response.
All of the read() methods, for both types of API endpoints, return the result of
a callable. By default, the callable returns a dictionary with two keys: metadata and
data. For response only tables, the metadata value will be the records from the
database as a pandas dataframe and the data will be `None.
Example – RegulatorAPI¶
regulator = RegulatorAPI()
result = await regulator.read()
result.get("metadata")
Record and File Endpoints¶
The record and file endpoints are the following:
-
CallingCardsBackground
-
Expression
-
PromoterSet
-
PromoterSetSig
-
RankResponse *
The default read() method is the same as the Records only Endpoint API classes.
However, there is an additional argument, retrieve_files which if set to True
will retrieve the file for which each record provides metadata. The return value of
read() is again a callable, and by default the data key will store a dictionary
where the keys correspond to the id column in the metadata.
# First, retrieve only the records -- you'll want to filter these results down before
# retrieving the files most likely
pss_api = PromoterSetSigAPI()
result = await pss_api.read()
result.get("metadata")
Filtering¶
All API classes have a params attribute which stores the filtering parameters
which will be applied to the HTTP requests.
pss_api.push_params({"regulator_symbol": "GZF3",
"workflow": "nf_core_callingcards_1_0_0",
"data_usable": "pass"})
pss_api.params
Retrieving files from a Records and Files Object¶
To retrieve files from a Records and Files endpoint object, do the following:
# note that retrieve_files is set to True
result = await pss_api.read(retrieve_files = True)
# the metadata slot is the same as before
result.get("metadata")
# but now the data slot is a dictionary where the `id` are keys and the values
# are the files parsed into pandas dataframes
result.get("data").get("6568")
Filtering on multiple items¶
Some filters, and eventually all, will accept multiple arguments as a comma separated string without spaces. For example:
pss_api.push_params({"regulator_symbol": "GZF3,RTG3"})
print(pss_api.params)
Parameters can be removed one by one
print(pss_api.params)
pss_api.pop_params('data_usable')
print(pss_api.params)
or cleared entirely
pss_api.pop_params(None)
print(pss_api.params)
Another example with Expression¶
This is used to get an expression_id to use in the RankResponseAPI() below
expression = ExpressionAPI()
expression.push_params({"regulator_symbol": "GZF3",
"lab": "mcisaac",
"time": "15"})
expression_res = await expression.read()
expression_res.get("metadata")
Rank Response¶
The rank response endpoint is slightly different than the others. It is implemented
asynchronously on the database side, and will run many tasks simultaneously. As such,
it uses submit() and retrieve() methods.
Additionally, it is a POST request and all parameters are passed in a json list. If you include, for a given dictionary, multiple items to promoetersetsig_ids and/or expression_ids, those datasets will be aggregated prior to calculating the rank response. The current parameters that may be passed for each are:
- promotersetsig_ids: a list of promotersetsig_ids. If more than 1, then the data will be aggregated prior to calcuating rank response
- expression_ids: a list of expression_ids. If more than 1, then the data will be aggregated prior to calcuating rank response
- expression_effect_colname: name of the column to use for the rank response effect
- expression_effect_threshold: The threshold to use on abs(effect) to label responsive/ unresponsive genes
- expression_pvalue_threshold: the threshold to use below which to label responsive/ unresponsive genes
- rank_bin_size: the size of the bins by which rank response is summarized.
- rank_by_binding_effect: if this is “true”, then rank by the binding effect first rather than pvalue. This is used for harbison_chip and mcisaac_oe
- summarize_by_rank_bin: if this is set to false, the unsummarized rank response (merged binding/response) is returned
rr_api = RankResponseAPI()
data = [
{
"promotersetsig_ids": ["5555"],
"expression_ids": ["2510"],
"rank_by_binding_effect": "true",
}
]
group_id = await rr_api.submit(post_dict=data)
result = await rr_api.retrieve(group_id)
result.get("metadata")
result.get("data").get(result.get("metadata").id[0])
df = result.get("data").get(result.get("metadata").id[0]).iloc[1:50,:]
# Plot
plt.figure(figsize=(10, 6))
plt.plot(df['rank_bin'], df['response_ratio'], marker='o')
plt.title('GZF3 - promotersetsig_id 6577. McIsaac 15 2510')
plt.xlabel('Rank Bin')
plt.ylabel('Response Ratio')
plt.title('Response Ratio vs Rank Bin')
plt.grid(True)
plt.show()
CallingCards Aggregated Data¶
For regulators which have data generated in the Brentlab and processed through the nf-core/callingcards:1.0.0 workflow, if that data has been manually (or eventually automatically) QC reviewed, and if there are at least 2 samples which are labeled as data_usable, then there will exist a BindingConcatenated record to which both a BindingManualQC record and a PromoterSetSig record are foreign keyed.
To view the set of samples for which there is aggregated data, you may do the following:
pss_api.pop_params(None)
pss_api.push_params({"source_name": "brent_nf_cc", "aggregated": "true"})
callingcards_aggregated_meta_res = await pss_api.read()
callingcards_aggregated_meta_df = callingcards_aggregated_meta_res.get("metadata")
callingcards_aggregated_meta_df
pss_api.push_params({"regulator_symbol": "GZF3"})
callingcards_res = await pss_api.read()
gzf3_callingcards_res = callingcards_res.get("metadata")
gzf3_callingcards_res
#gzf3_callingcards_res[~gzf3_callingcards_res.composite_binding_id.isna()]
data = [
{
"promotersetsig_ids": ["6873"],
"expression_ids": ["2510"],
}
]
group_id = await rr_api.submit(post_dict=data)
agg_result = await rr_api.retrieve(group_id)
agg_df = agg_result.get("data").get(agg_result.get("metadata").id[0]).iloc[1:50,:]
# Plot
plt.figure(figsize=(10, 6))
plt.plot(agg_df['rank_bin'], agg_df['response_ratio'], marker='o')
plt.title('Aggregated GZF3. McIsaac 15 2510')
plt.xlabel('Rank Bin')
plt.ylabel('Response Ratio')
plt.title('Response Ratio vs Rank Bin')
plt.grid(True)
plt.show()
Caveats¶
-
I have written the scripts to automatically check the redis queue for work and to both launch celery worker nodes, and kill them when they are finished. But, though they work if I run them manually, they have not worked when scheduled through a cronjob. I’ll work with Brian and Eric next week to figure out why.
-
I haven’t tested each of the endpoint APIs individually. Help is welcome.