Enabling realtime logging in Toil
Cactus Command: /private/home/hickey/dev/cactus.maf/venv-cactus.maf/bin/cactus-maf2bigmaf ./js-bb ./16-t2t-apes-2023v2.GCA_028885655.2.maf.gz ./16-t2t-apes-2023v2.GCA_028885655.2.bigmaf.bb --refGenome GCA_028885655.2 --halFile ./16-t2t-apes-2023v2.hal --logFile ./16-t2t-apes-2023v2.GCA_028885655.2.bigmaf.bb.log --batchLogsDir batch-logs-8apes --coordinationDir /data/tmp --batchSystem slurm
Cactus Commit: a8bd77e65d7f7c26fd7a6d69a110d1fe23b275c9
Importing ./16-t2t-apes-2023v2.GCA_028885655.2.maf.gz
Importing ./16-t2t-apes-2023v2.hal
Batch system does not support auto-deployment. The user script ModuleDescriptor(dirPath='/private/groups/cgl/hickey/work/cactus.maf/src', name='cactus.maf.cactus_maf2bigmaf', fromVirtualEnv=False) will have to be present at the same location on every worker.
Running Toil version 6.0.0-0e2a07a20818e593bfdfde3cc51ca4ad809fde96 on host phoenix.prism.
Starting real-time logging.
Issued job 'maf2bigmaf_workflow' kind-maf2bigmaf_workflow/instance-b16zzuuo v1 with job batch system ID: 1 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
1 jobs are running, 0 jobs are issued and waiting to run
2024-02-20 15:03:56.647525: Running the command: "mafToBigMaf"
2024-02-20 15:03:56.657133: Running the command: "bedToBigBed"
2024-02-20 15:03:56.664497: Running the command: "hgLoadMafSummary"
Issued job 'maf2bigmaf_chrom_sizes' kind-maf2bigmaf_chrom_sizes/instance-dbe3ne4o v1 with job batch system ID: 2 and disk: 32.1 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
Reading HAL file from job store to /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/a8b8/c19e/tmpw35cccuy/16-t2t-apes-2023v2.hal
Computing chromosome sizes
2024-02-20 15:06:23.970819: Running the command: "halStats /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/a8b8/c19e/tmpw35cccuy/16-t2t-apes-2023v2.hal --chromSizes GCA_028885655.2"
2024-02-20 15:06:23.984372: Successfully ran: "halStats /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/a8b8/c19e/tmpw35cccuy/16-t2t-apes-2023v2.hal --chromSizes GCA_028885655.2" in 0.0128 seconds and 8.5 Mi memory
2024-02-20 15:06:23.984499: Running the command: "halStats --genomes /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/a8b8/c19e/tmpw35cccuy/16-t2t-apes-2023v2.hal"
2024-02-20 15:06:24.038712: Successfully ran: "halStats --genomes /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/a8b8/c19e/tmpw35cccuy/16-t2t-apes-2023v2.hal" in 0.0537 seconds and 16.2 Mi memory
Got message from job at time 02-20-2024 15:06:24: Job used more disk than requested. For CWL, consider increasing the outdirMin requirement, otherwise, consider increasing the disk requirement. Job files/for-job/kind-maf2bigmaf_chrom_sizes/instance-dbe3ne4o/cleanup/file-2e8cc7357afa4144896bd9e13fe74bf2/stream used 100.00% disk (32.1 GiB [34454147072B] used, 32.1 GiB [34454088035B] requested).
Issued job 'maf2bigmaf_summary' kind-maf2bigmaf_summary/instance-dg2jvp42 v1 with job batch system ID: 3 and disk: 10.9 Gi, memory: 4.0 Gi, cores: 1, accelerators: [], preemptible: False
Issued job 'maf2bigmaf' kind-maf2bigmaf/instance-p27zsprz v1 with job batch system ID: 4 and disk: 38.3 Gi, memory: 4.0 Gi, cores: 1, accelerators: [], preemptible: False
Reading MAF file from job store to /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/01a5/1eef/tmpg3vk162q/16-t2t-apes-2023v2.GCA_028885655.2.maf.gz
Reading MAF file from job store to /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/2f43/877a/tmpmw59fs2e/16-t2t-apes-2023v2.GCA_028885655.2.maf.gz
2024-02-20 15:06:58.919561: Running the command: "bash -c set -eo pipefail && gzip -dc /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/01a5/1eef/tmpg3vk162q/16-t2t-apes-2023v2.GCA_028885655.2.maf.gz | mafDuplicateFilter -km - | mafToBigMaf GCA_028885655.2 stdin stdout | sort -k1,1 -k2,2n"
2024-02-20 15:06:58.919548: Running the command: "bash -c set -eo pipefail && gzip -dc /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/2f43/877a/tmpmw59fs2e/16-t2t-apes-2023v2.GCA_028885655.2.maf.gz | sed -f /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/2f43/877a/tmpmw59fs2e/genome_to_alnum.sed | mafDuplicateFilter -km - | hgLoadMafSummary -minSeqSize=1 -test g4 bigMafSummary stdin"
2024-02-20 15:29:48.358139: Successfully ran: "bash -c 'set -eo pipefail && gzip -dc /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/01a5/1eef/tmpg3vk162q/16-t2t-apes-2023v2.GCA_028885655.2.maf.gz | mafDuplicateFilter -km - | mafToBigMaf GCA_028885655.2 stdin stdout | sort -k1,1 -k2,2n'" in 1369.4373 seconds and 12.6 Mi memory
2024-02-20 15:29:48.358380: Running the command: "bedToBigBed -type=bed3+1 -as=/data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/01a5/1eef/tmpg3vk162q/bigMaf.as -tab /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/01a5/1eef/tmpg3vk162q/bigMaf.bed /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/01a5/1eef/tmpg3vk162q/GCA_028885655.2.chrom_sizes /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/01a5/1eef/tmpg3vk162q/16-t2t-apes-2023v2.GCA_028885655.2.bigmaf.bb"
2024-02-20 15:30:11.232497: Successfully ran: "bash -c 'set -eo pipefail && gzip -dc /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/2f43/877a/tmpmw59fs2e/16-t2t-apes-2023v2.GCA_028885655.2.maf.gz | sed -f /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/2f43/877a/tmpmw59fs2e/genome_to_alnum.sed | mafDuplicateFilter -km - | hgLoadMafSummary -minSeqSize=1 -test g4 bigMafSummary stdin'" in 1392.3118 seconds and 2.8 Gi memory
2024-02-20 15:30:11.233096: Running the command: "bash -c set -eo pipefail && cut -f2- bigMafSummary.tab | sed -f /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/2f43/877a/tmpmw59fs2e/alnum_to_genome.sed | sort -k1,1 -k2,2n"
2024-02-20 15:30:20.966977: Successfully ran: "bash -c 'set -eo pipefail && cut -f2- bigMafSummary.tab | sed -f /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/2f43/877a/tmpmw59fs2e/alnum_to_genome.sed | sort -k1,1 -k2,2n'" in 9.7332 seconds and 8.0 Mi memory
2024-02-20 15:30:20.967189: Running the command: "bedToBigBed -type=bed3+4 -as=/data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/2f43/877a/tmpmw59fs2e/mafSummary.as -tab /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/2f43/877a/tmpmw59fs2e/summary.bed /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/2f43/877a/tmpmw59fs2e/GCA_028885655.2.chrom_sizes /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/2f43/877a/tmpmw59fs2e/16-t2t-apes-2023v2.GCA_028885655.2.bigmaf.summary.bb"
2024-02-20 15:30:32.217996: Successfully ran: "bedToBigBed -type=bed3+4 -as=/data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/2f43/877a/tmpmw59fs2e/mafSummary.as -tab /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/2f43/877a/tmpmw59fs2e/summary.bed /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/2f43/877a/tmpmw59fs2e/GCA_028885655.2.chrom_sizes /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/2f43/877a/tmpmw59fs2e/16-t2t-apes-2023v2.GCA_028885655.2.bigmaf.summary.bb" in 11.2501 seconds and 23.8 Mi memory
2024-02-20 15:57:13.310388: Successfully ran: "bedToBigBed -type=bed3+1 -as=/data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/01a5/1eef/tmpg3vk162q/bigMaf.as -tab /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/01a5/1eef/tmpg3vk162q/bigMaf.bed /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/01a5/1eef/tmpg3vk162q/GCA_028885655.2.chrom_sizes /data/tmp/48e06dedbab65cb3aa8dad63288f9a6a/01a5/1eef/tmpg3vk162q/16-t2t-apes-2023v2.GCA_028885655.2.bigmaf.bb" in 1644.9511 seconds and 257.0 Mi memory
Got message from job at time 02-20-2024 15:57:27: Job used more disk than requested. For CWL, consider increasing the outdirMin requirement, otherwise, consider increasing the disk requirement. Job files/for-job/kind-maf2bigmaf/instance-p27zsprz/cleanup/file-a9a2c99d321e4b58a8ae97a04f436ba6/stream used 147.63% disk (56.5 GiB [60659699712B] used, 38.3 GiB [41088619180B] requested).
Finished toil run successfully.
Stopping real-time logging server.
Joining real-time logging server thread.
Exporting file:///private/groups/cgl/hickey/work/t2t-apes-release-2/temp3/16-t2t-apes-2023v2.GCA_028885655.2.bigmaf.bb
Exporting file:///private/groups/cgl/hickey/work/t2t-apes-release-2/temp3/16-t2t-apes-2023v2.GCA_028885655.2.bigmaf.summary.bb
Successfully deleted the job store: FileJobStore(/private/groups/cgl/hickey/work/t2t-apes-release-2/temp3/js-bb)
cactus-maf2bigmaf has finished after 3239.5970161220175 seconds