Difference between revisions of "Linux"

From www.wiki.chlamdb.ch
Jump to navigation Jump to search
(Created page with " * overview")
 
 
(18 intermediate revisions by 2 users not shown)
Line 1: Line 1:
  
−
* overview
+
==user/groups==
 +
 
 +
===create group and add users to the group===
 +
 
 +
<syntaxhighlight lang="Bash" line="line">
 +
sudo groupadd metagenlab
 +
 
 +
# a: append
 +
# G: group
 +
sudo usermod -a -G <group> <user>
 +
</syntaxhighlight>
 +
 
 +
*group file conf
 +
<syntaxhighlight lang="Bash" line="line">
 +
cat /etc/group
 +
</syntaxhighlight>
 +
 
 +
==HDD==
 +
 
 +
*liste des disques
 +
 
 +
<syntaxhighlight lang="Bash" line="line">
 +
lsblk -io KNAME,TYPE,SIZE,MODEL
 +
</syntaxhighlight>
 +
 
 +
==utilisation des diques==
 +
 
 +
<syntaxhighlight lang="Bash" line="line">
 +
df -h
 +
</syntaxhighlight>
 +
 
 +
==Docker==
 +
 
 +
*ubuntu network access problem
 +
 
 +
<syntaxhighlight lang="Bash" line="line">
 +
sudo mv /etc/resolv.conf{,.bak}
 +
sudo ln -s /etc/resolv.conf /run/systemd/resolve/resolv.conf
 +
</syntaxhighlight>
 +
 
 +
==Atom setup==
 +
 
 +
*[https://jstaf.github.io/2018/03/25/atom-ide.html atom - ipython - Hydrogen package]
 +
 
 +
 
 +
==NextFlow==
 +
 
 +
#http://nextflow-io.github.io/patterns/index.html#_get_process_work_directory
 +
#https://github.com/sanger-pathogens/companion/blob/master/annot.nf#L658
 +
#https://nextflow-io.github.io/nf-hack18/training.html#_basic_language_structures_and_commands
 +
 
 +
===Relauch killed jobs with increased ressources===
 +
 
 +
 
 +
<syntaxhighlight line="line">
 +
process foo {
 +
 
 +
    memory { 2.GB * task.attempt }
 +
    time { 1.hour * task.attempt }
 +
 
 +
    errorStrategy { task.exitStatus == 140 ? 'retry' : 'terminate' }
 +
    maxRetries 3
 +
 
 +
    script:
 +
    <your job here>
 +
 
 +
}
 +
</syntaxhighlight>
 +
 
 +
===split input into chunks===
 +
 
 +
<syntaxhighlight line="line">
 +
channel.buffer(size:3)
 +
</syntaxhighlight>
 +
===group channel output into a single list===
 +
 
 +
 
 +
<syntaxhighlight line="line">
 +
query = Channel.fromPath '*.fa'
 +
species = Channel.from 'human', 'cow', 'horse'
 +
 
 +
process blast {
 +
 
 +
input:
 +
    val species
 +
    file query
 +
 
 +
output:
 +
    set val(species), file('result') into blastOuts
 +
 
 +
 
 +
"blast -db nr -query $query" > result
 +
 
 +
}
 +
</syntaxhighlight>
 +
 
 +
In the above example a BLAST task is executed for each pair of species and query that are received. When the task completes a new tuple containing the value for species and the file result is sent to the blastOuts channel.
 +
 
 +
==make SSH listen to alternative port (eg 3389 rdp)==
 +
 
 +
*edit "/etc/ssh/sshd_config"
 +
*add "Port 3389"
 +
 
 +
==Bash error handling - exit code==
 +
 
 +
Capture exit code in piped commands<syntaxhighlight lang="bash">
 +
set -o pipefail
 +
</syntaxhighlight>Example: if grep cannot catch any match, it will return an exit code 1 (error) that will be catched if this option is set (default behaviour in snakemake rules)<syntaxhighlight lang="bash">
 +
fastaStat.py -f genome_fasta.fasta -l | grep contig_40 | sed 's//contig_/'
 +
</syntaxhighlight>Print the error code<syntaxhighlight lang="bash">
 +
echo $?
 +
</syntaxhighlight>If we do not whant the error to be catched, we can handle the exit code the following way:<syntaxhighlight>
 +
fastaStat.py -f genome_fasta.fasta -l | grep contig_40 || true | sed 's//contig_/'
 +
</syntaxhighlight><br />
 +
 
 +
* see http://redsymbol.net/articles/unofficial-bash-strict-mode/<br />

Latest revision as of 13:30, 20 February 2019

user/groups

create group and add users to the group

1sudo groupadd metagenlab
2
3# a: append
4# G: group
5sudo usermod -a -G <group> <user>
  • group file conf
1cat /etc/group

HDD

  • liste des disques
1lsblk -io KNAME,TYPE,SIZE,MODEL

utilisation des diques

1df -h

Docker

  • ubuntu network access problem
1sudo mv /etc/resolv.conf{,.bak}
2sudo ln -s /etc/resolv.conf /run/systemd/resolve/resolv.conf

Atom setup


NextFlow

  1. http://nextflow-io.github.io/patterns/index.html#_get_process_work_directory
  2. https://github.com/sanger-pathogens/companion/blob/master/annot.nf#L658
  3. https://nextflow-io.github.io/nf-hack18/training.html#_basic_language_structures_and_commands

Relauch killed jobs with increased ressources

process foo {

    memory { 2.GB * task.attempt }
    time { 1.hour * task.attempt }

    errorStrategy { task.exitStatus == 140 ? 'retry' : 'terminate' }
    maxRetries 3

    script:
    <your job here>

}

split input into chunks

channel.buffer(size:3)

group channel output into a single list

query = Channel.fromPath '*.fa'
species = Channel.from 'human', 'cow', 'horse'

process blast {

input:
    val species
    file query

output:
    set val(species), file('result') into blastOuts


"blast -db nr -query $query" > result

}

In the above example a BLAST task is executed for each pair of species and query that are received. When the task completes a new tuple containing the value for species and the file result is sent to the blastOuts channel.

make SSH listen to alternative port (eg 3389 rdp)

  • edit "/etc/ssh/sshd_config"
  • add "Port 3389"

Bash error handling - exit code

Capture exit code in piped commands

set -o pipefail

Example: if grep cannot catch any match, it will return an exit code 1 (error) that will be catched if this option is set (default behaviour in snakemake rules)

fastaStat.py -f genome_fasta.fasta -l | grep contig_40 | sed 's//contig_/'

Print the error code

echo $?

If we do not whant the error to be catched, we can handle the exit code the following way:

fastaStat.py -f genome_fasta.fasta -l | grep contig_40 || true | sed 's//contig_/'


Retrieved from "https://www.wiki.chlamdb.ch/index.php?title=Linux&oldid=192"