Enabling realtime logging in Toil
Cactus Command: /private/home/hickey/dev/cactus.maf/venv-cactus.maf/bin/cactus-maf2bigmaf ./js-bb ./16-t2t-apes-2023v2.GCA_028878055.2.maf.gz ./16-t2t-apes-2023v2.GCA_028878055.2.bigmaf.bb --refGenome GCA_028878055.2 --halFile ./16-t2t-apes-2023v2.hal --logFile ./16-t2t-apes-2023v2.GCA_028878055.2.bigmaf.bb.log --batchLogsDir batch-logs-8apes --coordinationDir /data/tmp --batchSystem slurm
Cactus Commit: a8bd77e65d7f7c26fd7a6d69a110d1fe23b275c9
Enabling realtime logging in Toil
Cactus Command: /private/home/hickey/dev/cactus.maf/venv-cactus.maf/bin/cactus-maf2bigmaf ./js-bb ./16-t2t-apes-2023v2.GCA_028878055.2.maf.gz ./16-t2t-apes-2023v2.GCA_028878055.2.bigmaf.bb --refGenome GCA_028878055.2 --halFile ./16-t2t-apes-2023v2.hal --logFile ./16-t2t-apes-2023v2.GCA_028878055.2.bigmaf.bb.log --batchLogsDir batch-logs-8apes --coordinationDir /data/tmp --batchSystem slurm
Cactus Commit: a8bd77e65d7f7c26fd7a6d69a110d1fe23b275c9
Importing ./16-t2t-apes-2023v2.GCA_028878055.2.maf.gz
Importing ./16-t2t-apes-2023v2.hal
Batch system does not support auto-deployment. The user script ModuleDescriptor(dirPath='/private/groups/cgl/hickey/work/cactus.maf/src', name='cactus.maf.cactus_maf2bigmaf', fromVirtualEnv=False) will have to be present at the same location on every worker.
Running Toil version 6.0.0-0e2a07a20818e593bfdfde3cc51ca4ad809fde96 on host phoenix.prism.
Starting real-time logging.
Issued job 'maf2bigmaf_workflow' kind-maf2bigmaf_workflow/instance-_xy2t_e5 v1 with job batch system ID: 1 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
1 jobs are running, 0 jobs are issued and waiting to run
2024-02-20 18:42:48.577854: Running the command: "mafToBigMaf"
2024-02-20 18:42:48.604160: Running the command: "bedToBigBed"
2024-02-20 18:42:48.622015: Running the command: "hgLoadMafSummary"
Issued job 'maf2bigmaf_chrom_sizes' kind-maf2bigmaf_chrom_sizes/instance-6stt9nh5 v1 with job batch system ID: 2 and disk: 32.1 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
Reading HAL file from job store to /data/tmp/e42549eb33955cf194aad91231110d9e/826d/80b9/tmpqe852p5j/16-t2t-apes-2023v2.hal
Computing chromosome sizes
2024-02-20 18:44:40.794533: Running the command: "halStats /data/tmp/e42549eb33955cf194aad91231110d9e/826d/80b9/tmpqe852p5j/16-t2t-apes-2023v2.hal --chromSizes GCA_028878055.2"
2024-02-20 18:44:40.832678: Successfully ran: "halStats /data/tmp/e42549eb33955cf194aad91231110d9e/826d/80b9/tmpqe852p5j/16-t2t-apes-2023v2.hal --chromSizes GCA_028878055.2" in 0.0373 seconds and 8.4 Mi memory
2024-02-20 18:44:40.832883: Running the command: "halStats --genomes /data/tmp/e42549eb33955cf194aad91231110d9e/826d/80b9/tmpqe852p5j/16-t2t-apes-2023v2.hal"
2024-02-20 18:44:40.887457: Successfully ran: "halStats --genomes /data/tmp/e42549eb33955cf194aad91231110d9e/826d/80b9/tmpqe852p5j/16-t2t-apes-2023v2.hal" in 0.0539 seconds and 16.2 Mi memory
Got message from job at time 02-20-2024 18:44:41: Job used more disk than requested. For CWL, consider increasing the outdirMin requirement, otherwise, consider increasing the disk requirement. Job files/for-job/kind-maf2bigmaf_chrom_sizes/instance-6stt9nh5/cleanup/file-bc70457b3de04fcbb5e537d25fbd0c91/stream used 100.00% disk (32.1 GiB [34454142976B] used, 32.1 GiB [34454088035B] requested).
Issued job 'maf2bigmaf' kind-maf2bigmaf/instance-u2cb09z_ v1 with job batch system ID: 3 and disk: 31.6 Gi, memory: 4.0 Gi, cores: 1, accelerators: [], preemptible: False
Issued job 'maf2bigmaf_summary' kind-maf2bigmaf_summary/instance-abt_8qa9 v1 with job batch system ID: 4 and disk: 9.0 Gi, memory: 4.0 Gi, cores: 1, accelerators: [], preemptible: False
Reading MAF file from job store to /data/tmp/e42549eb33955cf194aad91231110d9e/fa14/9dd6/tmpa5mgyi13/16-t2t-apes-2023v2.GCA_028878055.2.maf.gz
Reading MAF file from job store to /data/tmp/e42549eb33955cf194aad91231110d9e/f98c/21e4/tmp4qcb2ply/16-t2t-apes-2023v2.GCA_028878055.2.maf.gz
2024-02-20 18:45:05.807084: Running the command: "bash -c set -eo pipefail && gzip -dc /data/tmp/e42549eb33955cf194aad91231110d9e/f98c/21e4/tmp4qcb2ply/16-t2t-apes-2023v2.GCA_028878055.2.maf.gz | mafDuplicateFilter -km - | mafToBigMaf GCA_028878055.2 stdin stdout | sort -k1,1 -k2,2n"
2024-02-20 18:45:05.807293: Running the command: "bash -c set -eo pipefail && gzip -dc /data/tmp/e42549eb33955cf194aad91231110d9e/fa14/9dd6/tmpa5mgyi13/16-t2t-apes-2023v2.GCA_028878055.2.maf.gz | sed -f /data/tmp/e42549eb33955cf194aad91231110d9e/fa14/9dd6/tmpa5mgyi13/genome_to_alnum.sed | mafDuplicateFilter -km - | hgLoadMafSummary -minSeqSize=1 -test g10 bigMafSummary stdin"
2024-02-20 19:05:01.721658: Successfully ran: "bash -c 'set -eo pipefail && gzip -dc /data/tmp/e42549eb33955cf194aad91231110d9e/fa14/9dd6/tmpa5mgyi13/16-t2t-apes-2023v2.GCA_028878055.2.maf.gz | sed -f /data/tmp/e42549eb33955cf194aad91231110d9e/fa14/9dd6/tmpa5mgyi13/genome_to_alnum.sed | mafDuplicateFilter -km - | hgLoadMafSummary -minSeqSize=1 -test g10 bigMafSummary stdin'" in 1195.9131 seconds and 2.6 Gi memory
2024-02-20 19:05:01.722030: Running the command: "bash -c set -eo pipefail && cut -f2- bigMafSummary.tab | sed -f /data/tmp/e42549eb33955cf194aad91231110d9e/fa14/9dd6/tmpa5mgyi13/alnum_to_genome.sed | sort -k1,1 -k2,2n"
2024-02-20 19:05:11.740569: Successfully ran: "bash -c 'set -eo pipefail && cut -f2- bigMafSummary.tab | sed -f /data/tmp/e42549eb33955cf194aad91231110d9e/fa14/9dd6/tmpa5mgyi13/alnum_to_genome.sed | sort -k1,1 -k2,2n'" in 10.0178 seconds and 7.9 Mi memory
2024-02-20 19:05:11.740779: Running the command: "bedToBigBed -type=bed3+4 -as=/data/tmp/e42549eb33955cf194aad91231110d9e/fa14/9dd6/tmpa5mgyi13/mafSummary.as -tab /data/tmp/e42549eb33955cf194aad91231110d9e/fa14/9dd6/tmpa5mgyi13/summary.bed /data/tmp/e42549eb33955cf194aad91231110d9e/fa14/9dd6/tmpa5mgyi13/GCA_028878055.2.chrom_sizes /data/tmp/e42549eb33955cf194aad91231110d9e/fa14/9dd6/tmpa5mgyi13/16-t2t-apes-2023v2.GCA_028878055.2.bigmaf.summary.bb"
2024-02-20 19:05:20.603795: Successfully ran: "bedToBigBed -type=bed3+4 -as=/data/tmp/e42549eb33955cf194aad91231110d9e/fa14/9dd6/tmpa5mgyi13/mafSummary.as -tab /data/tmp/e42549eb33955cf194aad91231110d9e/fa14/9dd6/tmpa5mgyi13/summary.bed /data/tmp/e42549eb33955cf194aad91231110d9e/fa14/9dd6/tmpa5mgyi13/GCA_028878055.2.chrom_sizes /data/tmp/e42549eb33955cf194aad91231110d9e/fa14/9dd6/tmpa5mgyi13/16-t2t-apes-2023v2.GCA_028878055.2.bigmaf.summary.bb" in 8.8623 seconds and 22.9 Mi memory
2024-02-20 19:05:27.153200: Successfully ran: "bash -c 'set -eo pipefail && gzip -dc /data/tmp/e42549eb33955cf194aad91231110d9e/f98c/21e4/tmp4qcb2ply/16-t2t-apes-2023v2.GCA_028878055.2.maf.gz | mafDuplicateFilter -km - | mafToBigMaf GCA_028878055.2 stdin stdout | sort -k1,1 -k2,2n'" in 1221.3449 seconds and 12.2 Mi memory
2024-02-20 19:05:27.153446: Running the command: "bedToBigBed -type=bed3+1 -as=/data/tmp/e42549eb33955cf194aad91231110d9e/f98c/21e4/tmp4qcb2ply/bigMaf.as -tab /data/tmp/e42549eb33955cf194aad91231110d9e/f98c/21e4/tmp4qcb2ply/bigMaf.bed /data/tmp/e42549eb33955cf194aad91231110d9e/f98c/21e4/tmp4qcb2ply/GCA_028878055.2.chrom_sizes /data/tmp/e42549eb33955cf194aad91231110d9e/f98c/21e4/tmp4qcb2ply/16-t2t-apes-2023v2.GCA_028878055.2.bigmaf.bb"
2024-02-20 19:32:44.893114: Successfully ran: "bedToBigBed -type=bed3+1 -as=/data/tmp/e42549eb33955cf194aad91231110d9e/f98c/21e4/tmp4qcb2ply/bigMaf.as -tab /data/tmp/e42549eb33955cf194aad91231110d9e/f98c/21e4/tmp4qcb2ply/bigMaf.bed /data/tmp/e42549eb33955cf194aad91231110d9e/f98c/21e4/tmp4qcb2ply/GCA_028878055.2.chrom_sizes /data/tmp/e42549eb33955cf194aad91231110d9e/f98c/21e4/tmp4qcb2ply/16-t2t-apes-2023v2.GCA_028878055.2.bigmaf.bb" in 1637.7388 seconds and 220.5 Mi memory
Got message from job at time 02-20-2024 19:33:22: Job used more disk than requested. For CWL, consider increasing the outdirMin requirement, otherwise, consider increasing the disk requirement. Job files/for-job/kind-maf2bigmaf/instance-u2cb09z_/cleanup/file-bc73b51108534c79b124532fc0797ae7/stream used 167.73% disk (53.1 GiB [56995201024B] used, 31.6 GiB [33981188122B] requested).
Finished toil run successfully.
Stopping real-time logging server.
Joining real-time logging server thread.
Exporting file:///private/groups/cgl/hickey/work/t2t-apes-release-2/temp3/16-t2t-apes-2023v2.GCA_028878055.2.bigmaf.bb
Exporting file:///private/groups/cgl/hickey/work/t2t-apes-release-2/temp3/16-t2t-apes-2023v2.GCA_028878055.2.bigmaf.summary.bb
Successfully deleted the job store: FileJobStore(/private/groups/cgl/hickey/work/t2t-apes-release-2/temp3/js-bb)
cactus-maf2bigmaf has finished after 3088.5909621779574 seconds