www.wiki.chlamdb.ch - User contributions [en]

ChlamDB

2020-03-10T17:48:11Z

Tpillone: /* setup X server pour ete3 */

==Mise en ligne de la base de donnée Chlamdb==

===Page d’accueil===

*mettre un nouvel onglet avec qq infos
*écrire une page tuto avec fonctions de base
*exporter le fichier avec environnementale conda nécessaire pour faire tourner la base de donnée

===Setup chlamdb===

[[old setup]]

[[new setup]]

===Update===
- avril 2019
{| class="wikitable sortable"
|+
!
!N genomes
!N annotated Genbank
!N annotated RS
!N complete genomes
|-
|'''Planctomycetes'''
|600
|423
|60
|20
|-
|'''Chlamydiae'''
|463
|392
|401
|165
|-
|'''Ca. Omnitrophica'''
|172
|137
|4
|0
|-
|'''Kiritimatiellaeota'''
|13
|7
|1
|1
|-
|'''Lentisphaerae'''
|62
|29
|4
|1
|-
|'''Verrucomicrobia'''
|930
|555
|158
|34
|-
|'''TOTAL'''
|'''2249'''
|'''1443'''
|'''628'''
|'''221'''
|}

===Setup environnement fonctionnel===

====Conda====

name: chlamdb
channels:
- r
- etetoolkit
- bioconda
- conda-forge
- defaults
dependencies:
- python=3.5.6
- django=2.1.1
- django-extensions
- bibtexparser
- cairo
- r-cairo
- r-vegan
- r-gplots
- r-gdtools
- r-svglite
- pandas
- qtconsole
- ete3
- ete_toolchain
- pyqt=4.11.4
- matplotlib
- scipy
- pylibmc
- simplejson
- dill
- gunicorn
- numpy
- pyqt
- lxml
- conda-build
- reportlab
- biopython
- django-crispy-forms
- mysqlclient
- pip:
- cython
- django-citations

====packages R absents de conda à installer à la main====

- genoPlotR
- flextable

====modules python a ajouter à la main====
<syntaxhighlight lang="Bash" line="line">
conda develop /home/ubuntu/dev/orthomcl/
conda develop /home/ubuntu/dev/plots/
conda develop /home/ubuntu/dev/biosqldb/
conda develop /home/ubuntu/dev/utilitaires/
conda develop /home/ubuntu/dev/phylo_tree_display/
conda develop /home/ubuntu/dev/TPutils/
conda develop /home/ubuntu/dev/network_d3/
</syntaxhighlight>
====setup X server pour ete3====

<syntaxhighlight lang="Bash" line="line">
apt-get install xserver-xorg xdm xfonts-base xfonts-100dpi xfonts-75dpi
</syntaxhighlight>

<syntaxhighlight lang="Bash" line="line">
In our example, edit the /etc/X11/xdm/xdm-config file and set following values:

DisplayManager*authorize: false
DisplayManager*authComplain: false
</syntaxhighlight>

<syntaxhighlight lang="Bash" line="line">
/etc/init.d/xdm restart
</syntaxhighlight>

<syntaxhighlight lang="Bash" line="line">
export DISPLAY=':0'
</syntaxhighlight>

Might be necessary to update the Xservers file as well (/etc/X11/xdm/Xservers)

<syntaxhighlight lang="Bash" line="1">
#:0 local /usr/X11R6/bin/X -dpi 100 -nolisten tcp
:0 local /usr/X11R6/bin/X -dpi 100
</syntaxhighlight>
==setup https==

===setup certificate and key with letsencrypt===

<syntaxhighlight lang="Bash" line="line">
systemctl stop nginx
letsencrypt certonly
systemctl start nginx
</syntaxhighlight>

===update config===

<syntaxhighlight lang="Bash" line="line">
upstream monsite {
# Adresse du serveur wsgi, par exemple gunicorn
server localhost:8001;
}

# HTTP Request will be Redirected to the HTTPS
server {
listen 80;
server_name chlamdb.ch;
return 301 https://$server_name$request_uri;
}

server {
listen 443 ssl;
server_name chlamdb.ch;

ssl_certificate /etc/letsencrypt/live/chlamdb.ch/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/chlamdb.ch/privkey.pem;

access_log /webapps/logs/GEN.access.log;
error_log /webapps/logs/GEN.error.log;

location / {
#include fastcgi_params;
proxy_pass http://monsite;
proxy_redirect off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}

location /assets {
autoindex on;
alias /home/ubuntu/dev/chlamdb/assets/;
}
}

</syntaxhighlight>

=setup celery=

- install celery, django-celery
- install rabbitmq
<syntaxhighlight lang="Bash" line="line">
sudo apt-get install rabbitmq-server
sudo systemctl enable rabbitmq-server
sudo systemctl start rabbitmq-server
</syntaxhighlight>

- setup user and host

<syntaxhighlight lang="Bash" line="line">
sudo rabbitmqctl add_vhost my_vhost
sudo rabbitmqctl add_user chlamdb estrella3
sudo rabbitmqctl set_permissions -p my_vhost chlamdb "." "." ".*"
</syntaxhighlight>

- launch celery workers

<syntaxhighlight lang="Bash" line="line">
celery -A chlamdb.celeryapp worker -l info
</syntaxhighlight>

=setup on vital-it=

==ete3==

work with the following variable exported:

<syntaxhighlight lang="Bash" line="line">
export QT_QPA_PLATFORM=offscreen
</syntaxhighlight>

JBrowse

2019-09-06T13:26:10Z

Tpillone:

==folder structure==
 

#folder raw data: /var/www/html/jbrowse/sample_data/raw/
#folder configuration files: /var/www/html/jbrowse/sample_data/json/

We have to create a new folder for each dataset:

<syntaxhighlight lang="bash">
mkdir /var/www/html/jbrowse/sample_data/raw/MTB
mkdir /var/www/html/jbrowse/sample_data/json/MTB
</syntaxhighlight>

==Add reference from fasta==

<syntaxhighlight>
cd /var/www/html/jbrowse/sample_data/json/MTB
prepare-refseqs.pl --fasta NC_000962.fna
</syntaxhighlight> 

*This will create
**/var/www/html/jbrowse/sample_data/json/MTB/tracks.conf
**/var/www/html/jbrowse/sample_data/json/MTB/trackList.json
**/var/www/html/jbrowse/sample_data/json/MTB/seq

==Add bam track files==

*Add sorted bam file in "/var/www/html/jbrowse/sample_data/raw/MTB"
*Index bam file
<syntaxhighlight>
cd /var/www/html/jbrowse/sample_data/raw/MTB
samtools index 538048.bam

</syntaxhighlight>

*Add bam track to tracks.conf
<syntaxhighlight>
emacs -nw /var/www/html/jbrowse/sample_data/json/MTB/tracks.conf
</syntaxhighlight>Content:<syntaxhighlight lang="json">
# make a section for the new track called tracks.(unique name)
[ tracks.my-bam-track ]

# settings for what data is shown in the track
storeClass = JBrowse/Store/SeqFeature/BAM
urlTemplate = ../../raw/MTB/538048.bam
baiUrlTemplate = ../../raw/MTB/538048.bam.bai

type = JBrowse/View/Track/Alignments2
key = BAM alignments from sample XYZ
renderPrettyAlignments = true

[ tracks.bam-coverage ]

storeClass = JBrowse/Store/SeqFeature/BAM
urlTemplate = ../../raw/MTB/538048.bam
baiUrlTemplate = ../../raw/MTB/538048.bam.bai

type = JBrowse/View/Track/SNPCoverage
</syntaxhighlight>

==Add gene track==
<syntaxhighlight lang="bash">
# convert gbk to gff
gbk2gff.py -i genome_gbwithparts.gbk -o genome_gbwithparts.gff

# add track
flatfile-to-json.pl --gff genome_gbwithparts.gff --type CDS --trackLabel ORFs --out /var/www/html/jbrowse/sample_data/json/MTB --nameAttributes locus_tag
</syntaxhighlight>

==Add locus tags index==
 <syntaxhighlight lang="bash">
cd /var/www/html/jbrowse/sample_data/raw/MTB
generate-names.pl --verbose --out /var/www/html/jbrowse/sample_data/json/MTB
</syntaxhighlight>

This will create

*/var/www/html/jbrowse/sample_data/json/MTB/names

== Access browser ==

[http://127.0.0.1/jbrowse/index.html?data=sample_data%2Fjson%2FMTB http://127.0.0.1/jbrowse/index.html?data=sample_data/json/MTB]

JBrowse

2019-09-06T13:25:06Z

Tpillone: /* Add locus tags index */

==folder structure==
 

#folder raw data: /var/www/html/jbrowse/sample_data/raw/
#folder configuration files: /var/www/html/jbrowse/sample_data/json/

We have to create a new folder for each dataset:

<syntaxhighlight lang="bash">
mkdir /var/www/html/jbrowse/sample_data/raw/MTB
mkdir /var/www/html/jbrowse/sample_data/json/MTB
</syntaxhighlight>

==Add reference from fasta==

<syntaxhighlight>
cd /var/www/html/jbrowse/sample_data/json/MTB
prepare-refseqs.pl --fasta NC_000962.fna
</syntaxhighlight> 

* This will create
** /var/www/html/jbrowse/sample_data/json/MTB/tracks.conf
** /var/www/html/jbrowse/sample_data/json/MTB/trackList.json
** /var/www/html/jbrowse/sample_data/json/MTB/seq

==Add bam track files==

*Add sorted bam file in "/var/www/html/jbrowse/sample_data/raw/MTB"
*Index bam file
<syntaxhighlight>
cd /var/www/html/jbrowse/sample_data/raw/MTB
samtools index 538048.bam

</syntaxhighlight>

*Add bam track to tracks.conf
<syntaxhighlight>
emacs -nw /var/www/html/jbrowse/sample_data/json/MTB/tracks.conf
</syntaxhighlight>Content:<syntaxhighlight lang="json">
# make a section for the new track called tracks.(unique name)
[ tracks.my-bam-track ]

# settings for what data is shown in the track
storeClass = JBrowse/Store/SeqFeature/BAM
urlTemplate = ../../raw/MTB/538048.bam
baiUrlTemplate = ../../raw/MTB/538048.bam.bai

type = JBrowse/View/Track/Alignments2
key = BAM alignments from sample XYZ
renderPrettyAlignments = true

[ tracks.bam-coverage ]

storeClass = JBrowse/Store/SeqFeature/BAM
urlTemplate = ../../raw/MTB/538048.bam
baiUrlTemplate = ../../raw/MTB/538048.bam.bai

type = JBrowse/View/Track/SNPCoverage
</syntaxhighlight>

==Add gene track==
<syntaxhighlight lang="bash">
# convert gbk to gff
gbk2gff.py -i genome_gbwithparts.gbk -o genome_gbwithparts.gff

# add track
flatfile-to-json.pl --gff genome_gbwithparts.gff --type CDS --trackLabel ORFs --out /var/www/html/jbrowse/sample_data/json/MTB --nameAttributes locus_tag
</syntaxhighlight>

==Add locus tags index==
 <syntaxhighlight lang="bash">
cd /var/www/html/jbrowse/sample_data/raw/MTB
generate-names.pl --verbose --out /var/www/html/jbrowse/sample_data/json/MTB
</syntaxhighlight>

This will create

* /var/www/html/jbrowse/sample_data/json/MTB/names

Tpillone:

=Tools=

{| class="wikitable sortable"
|-
!Software!!Topic
|-
|[[bwa]]||Mapping
|-
|[[centrifuge]]||Taxonomic classification of nucleotide sequences
|-
|[[kaiju]]||Taxonomic classification of amino acid sequences
|}

==Pipelines Snakemake==
 

#https://github.com/metagenome-atlas/atlas

 
=Links=

#https://github.com/snayfach/MIDAS
#https://github.com/ncbi/sra-tools/wiki/HowTo:-fasterq-dump
#http://calcul.math.cnrs.fr/IMG/pdf/presentation.pdf
#https://github.com/gjospin/PhyloSift
#http://hannonlab.cshl.edu/fastx_toolkit/commandline.html
#https://github.com/JoseBlanca/seq_crumbs
#https://github.com/soedinglab/MMseqs2/wiki#install-static-linux-version
#https://github.com/SaierLaboratory/BioVx/blob/master/scripts/gblast3.py
#https://ccb.jhu.edu/software/bracken/
#http://motu-tool.org/
#https://www.ebi.ac.uk/metagenomics/
#https://www.nature.com/articles/d41586-018-07196-1
#https://www.taxonomer.com/
#http://segatalab.cibio.unitn.it/tools/
#https://academic.oup.com/nar/article/45/2/e7/2953304
#https://www.nature.com/articles/nmeth.3802
#https://research.bioinformatics.udel.edu/peptidematch/index.jsp
#http://lab.dessimoz.org/blog/2018/10/01/human-plant-orthologs-p2
#http://merenlab.org/software/anvio/
#http://www.bigsi.io

=Biblio=

#https://www.nature.com/articles/s41586-018-0703-0.epdf?shared_access_token=gm4GZr3P2ZZxDN8RGNy7LdRgN0jAjWel9jnR3ZoTv0N9OWpzII7VEgb2rgQ968D8MuuDkP3J2YZRratQgCRBTWXIBHcUKnD0HqETn9DHIBBPy5h4d7-POMKMRzGuhjJ8D3O4jLPrj4JrH7tG6gNQ6GQV77d_1eo568aMvymB-GU%3D
#https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6090853/
#https://watermark.silverchair.com/msy123.pdf?token=AQECAHi208BE49Ooan9kkhW_Ercy7Dm3ZL_9Cf3qfKAc485ysgAAAkgwggJEBgkqhkiG9w0BBwagggI1MIICMQIBADCCAioGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMHUixy6gkLUA3-pjLAgEQgIIB-y5PdqIXTXk8tHL2nlaGIkx8nd-e6zdeifITsMs_WhG6Ri-iq0a_7w4lbB8DB8sTYrQefo0r0t5kda2AcRpaljP6V9pazbnx0nI_8qYqSUHlwazmGiXp2d01On_dz0CnsWnU3_9lBG8V8BWqxFLxtoDGcpDEi7QuEGOfNqXtvFLIzjDktbIORSiGSbpfqHo43UXby9IIDgnh8QmKoDm6zegZ_l2xVsPFJX1g_TvJToF9e0ORWuexNBGE5yMpKIN26aVse4BeS5lnrW-TXkLAvLh7eKNSMYznL4FK4gklCDFvDf6dAESrCgIEjbmBRGR5R6zTh3Ldw5PRHdLSquUXZ3sxqAGYZkFq2msd86fxaBqwE5UhJQHbNDsR1UD6CdfI7Dr12rCaqJgWyEHjwU9d-TNvZpu6buSBESQAgOjwEB7fWZc2MdyijGfD8dPYdkbsHYwiCjVxiSXXlIVehpRo8rNz9T9k7z38WYSLOjnELMRDl1NTerZyVRz2ixGQzLkXz7D0_iGvfgh6LIlvtYnfAQZmX1hBDfaxTjMbQwVJjvT8DS3dmXgYoc2J8sdEpi2dUwb6gt3XUbxbskvaYfDXo0JeQTPq5wftD54Iwp74VdYw5gHssdtFkbLLsh83ao2dtNDnlu0Uc2QB7gQ_TYRBmYPB3UzhkuTgaHOy0g
#https://research.pasteur.fr/fr/project/population-genomics/
#https://medium.com/@infoecho/the-end-of-a-chapter-a-memoir-on-bioinformatics-development-work-for-single-molecule-sequencing-19455c288280
#https://watermark.silverchair.com/bbw020.pdf?token=AQECAHi208BE49Ooan9kkhW_Ercy7Dm3ZL_9Cf3qfKAc485ysgAAAlowggJWBgkqhkiG9w0BBwagggJHMIICQwIBADCCAjwGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMq2CbkYjDYcR3lshgAgEQgIICDcfmCZSqsrJPkLxOgW2ftW4xru0PuqsKrx4TG05vnF9dedAd9P8Ya8xn3gjY7dsC1XtnUoxhj0J9x9gXI3-N0KI1oVSxy5BpUX6f40iJS9L5tBSv2FAAK108r983tECXELKnATZj1HQYtz9RCkasj3tsA5QxQULDHcgj195vsn6fGulC3StrzNnbEu4oZuk6e2LAEQJbSYHE8UsDw2yngo0s80FaHyWhduJynK_NsdIwt2aVXV2HB7BZG0uTYYQU0-R00wX75wjDg5hrdW3lsYIvq87XlmyKAkug_vCHyh0rOURVZXZXRlqd5uxj4zgb1jgMU5ue18RWxe00Rw_ZoHnaVIUSbM5E7xLEUA6hFS2Gqmoe1_j5Gq_b21dWVi3exT2TpIrzc2GpZbw3a5IegaaI9yTEICauJ21UhzPXNzCmR9aLqc45kDj5Kahhc2pV3sNMANyReqMuQc9SREa7BVA58L-FVEVOVYzREG837X29bRzEIPK1SMtE-n2f4WXNOdDK--M7jumgKJubztly_cNZfnC_hfGYBRNmBf4dWDXuFE50RfWdpnVkYMtXvWGrk5rJ6tpwvR490vhVKCWoaWLFm00R9yI9sTiReSUapvGaxyw77Uq1rdoDcjAgcETYlXUpvqC7Q2GsAXtJiVi7NMo-XZz-sDcY9hOfElIMiQCEk7CwWfQZ6X6aMEakhg

#https://www.ncbi.nlm.nih.gov/pmc/articles/PMC165453/

=Estimation du génome sur la base des kmer=

#http://kmergenie.bx.psu.edu/
#https://khmer-recipes.readthedocs.io/en/latest/003-estimate-genome-size/index.html
#https://khmer-recipes.readthedocs.io/en/latest/005-estimate-total-genome-size/
#https://khmer-recipes.readthedocs.io/en/latest/
#http://koke.asrc.kanazawa-u.ac.jp/HOWTO/kmer-genomesize.html

=Meteganomic: taxonomic assignation=

#http://porecamp.github.io/2017/metagenomics.html
#https://ccb.jhu.edu/software/centrifuge/manual.shtml

=Networks=

#youtube network

#http://www.yasiv.com/youtube#?Search%3Fq=write%20in%20c&q=N'to%20-%20La%20Cl%C3%A9%20Des%20Champs
#https://github.com/anvaka/VivaGraphJS

=maps=

#word density maps:

=Installation de ScanIndel (+BLAT)=

- https://github.com/cauyrd/ScanIndel
- via conda
- donwloder executables manquant: http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/

TODO

2019-02-20T13:46:16Z

Tpillone:

==A lire==

====Virual envs/Pipelines====

*https://www.taniarascia.com/what-are-vagrant-and-virtualbox-and-how-do-i-use-them/

=== Tools ===

* http://wwwuser.gwdg.de/~compbiol/molbio_course/2018/worksheet-day1.pdf

*

===Biology===

*https://lcbb.epfl.ch/compbio14/
*https://viral-ngs.readthedocs.io/en/v1.1.2/index.html